Optimizations for live event, real-time, 3d object tracking

ABSTRACT

A system for automatically providing event non-video information usable for indexing event video information, where the event takes place at an event location for a duration of event time. The system includes a non-video information system for receiving or determining event non-video information and storing the non-video information in a non-video dataset, a video information system for receiving event video information captured by one or more filming cameras and storing the video information in a video dataset, and a video information retrieval system for randomly accessing the video dataset, where the random access includes: (a) using event non-video information to first determine an event time or duration of event time, and (b) using the determined event time or duration of event time to selectively access, retrieve, or provide event video information including one or more video images of the event.

RELATED APPLICATIONS

The present invention is a continuation of U.S. patent application Ser.No. 12/287,339 filed Oct. 8, 2008 (allowed); which was a divisional ofU.S. patent application Ser. No. 10/006,444 filed Nov. 20, 2001, andissued as U.S. Pat. No. 7,483,049 on Jan. 27, 2009; which was acontinuation-in-part of U.S. patent application Ser. No. 09/510,922filed on Feb. 22, 2000, and issued as U.S. Pat. No. 6,707,487 on Mar.16, 2004; which was a continuation-in-part of U.S. patent applicationSer. No. 09/197,219 filed on Nov. 20, 1998, and issued as U.S. Pat. No.6,567,116 on May 20, 2003; with all of the foregoing applications andpatents incorporated in this application by reference.

TECHNICAL FIELD

The present invention relates to machine vision systems for tracking themovement of multiple objects within a predefined area or volume.

BACKGROUND OF THE INVENTION

Several systems currently exist in the commercial marketplace fortracking the movement of one or more objects within a limited area. Suchsystems are routinely used to follow the movement of human subjects fora range of purposes including medical analysis, input for lifelikeanimation, as well as sports measurement. The following companiesprovide machine vision-based motion analysis systems:

-   -   Motion Analysis Corporation and their HiRES 3D system;    -   Vicon with their Vicon 250 and 512 3D motion analysis systems;    -   Ariel Dynamics, Inc. with their APAS system;    -   Charnwood Dynamics with their CODA motion analysis system;    -   Peak Performance Inc. with their Maus system;    -   Biogesta with their SAGA-3 RT System;    -   Elite with their ELITEPlus Motion Analyser System;    -   Northern Digital with their Optotrak and Polaris systems, and    -   Qualisys with their ProReflex system.

Each of these systems, which are capable of working in real time andcreating three-dimensional (3D) tracking information, employ a system ofmarkers to be placed upon the object(s) to be tracked. The markersthemselves are followed by an overlapping configuration of trackingcameras. As image information is analyzed from each camera'stwo-dimensional (2D) view, it is combined to create the 3D coordinatesof each marker as that marker moves about in the designated trackingvolume. Based upon the detected marker 3D locations as well as thepre-known relationship between the markers and the objects, each systemis then able to “re-assemble” any given object's 3D movement. All of thesystems share at least portions of the following common attributes:

1—All of the cameras that are used to view the marker and thereforeobject movement are pre-placed in fixed strategic locations designed tokeep the entire tracking volume in view of two or more cameras.

2—Each camera is designed to capture a unique 2D view for a fixedportion of the tracking volume. The entire set of captured 2Dinformation is combined by the system to create the 3D informationconcerning all markers and therefore objects.

3—For a marker to be located in the local (X, Y, Z) coordinate systemduring any given instant, it must be visible to at least two cameraswithin that instant.

4—They use a single tracking energy that is either from the visiblespectrum (such as red light) or infrared (IR).

5—They add additional tracking energy in the form of LED-based ringlights attached to the tracking cameras.

6—They place special retroreflective markers on the objects to betracked. These retroreflectors reflect a broad spectrum of energyincluding visible and IR light.

7—The markers do not comprise any special encoding and are most oftenidentical in size and shape. Typical shapes are a rectangle, a circle,or a sphere.

8—They use the unique positional combination (i.e., the measuredphysical space relationship of the markers placed upon the object) toencode that object's identity. Hence, no two objects can have the sameor a substantially similar positional combination of markers placed uponthem. This “constellation” of markers covers the majority of theobject's surface volume and as such requires that the entire volumeremain substantially in view at all times.

9—They determine, confirm, or both determine and confirm the identity ofeach object simultaneously with the tracking of the objects as they movethroughout the entire tracking volume.

10—After the cameras have been placed in their fixed positions, theycalibrate the system prior to tracking by moving a special calibrationtool throughout the combined views of all cameras. The calibration toolconsists of two or more markers that are at a pre-known distance fromeach other. Once the calibration has been completed, none of the camerasmay be moved before or during actual object tracking.

Each of these systems shares many drawbacks, especially when consideredfor use in a live sporting environment such as a sporting contest. Someof these drawbacks are as follows:

1—All of the tracking cameras must be set into fixed positions and thenpre-calibrated prior to actual live tracking. This requirement precludesthe use of automatic pan, tilt, and zoom cameras to collect additionalinformation as directed by the system in anticipation of markerinclusions.

2—Each camera is positioned to have a unique and substantiallyperspective view of a given portion of the tracking volume. Eachcamera's perspective view contains a significant depth of field. Anygiven object traveling throughout this depth of field will be seen withsubstantially different resolutions depending upon whether the object isat the nearest or farthest point with respect to a given camera.Therefore, the system experiences a non-uniform resolution per objectthroughout the entire tracking volume. This non-uniform resolutionaffects at least the ease with which the system may be scaled up tocover larger and larger tracking volumes using a consistent cameraarrangement.

3—If the tracking energy is red light, any human observers will also seethe illuminated markers if they are within the narrow retroreflectedcone of light.

4—If the tracking energy is red light, then the system is susceptible toreflections of red light caused by pre-existing lighting in the visiblespectrum that may be reflected from red colored portions of the trackedobjects or the tracking volume itself.

5—In order to reduce unwanted reflections when working with visible redlight, the systems typically cover the objects in darker material andplace black matting on the movement surface to help reduce unwantedreflections that become system noise. These techniques are notappropriate for a “live” environment.

6—If the tracking energy is IR, then the light sources employed onlyemit IR without any additional visible light. The additional visiblelight would normally act as an indicator to a human observer that thelight is on, naturally causing them not to stare for prolonged periods.Continued exposure to any high-intensity energy including IR light candamage the retina of the eye.

7—When working with IR, these systems do not employ IR absorbentcompounds to be placed upon the objects and tracking volume backgroundsurfaces before any markers are attached as a means of reducing unwantedreflections that become system noise.

8—Because the retroreflective markers work across a broad spectrum, theywill reflect any visible energy, not just the chosen emitted trackingenergy whether that be red light or IR. As such, they will for instanceretroreflect any pre-existing lighting or portable lighting such ascamera flashes that are typically used by human observers in a liveenvironment.

9—Given that the preferred spherical markers do have an appreciablesize, they are limited in the number of places that they can be placedupon the objects, especially in a live environment. Due to their size,they are impractical for use in live sporting contests especiallycontact sports such as ice hockey where they may become dislodged duringnormal play.

10—When the systems are used to track more and more objects, each withmany markers, more and more instances arise when not all markers are inview of at least two cameras or in some cases in view of any camera.This is referred to as “inclusions” and also affects the ability of thesystem to accurately identify a given object since its identity isencoded in the unique “constellation” of the markers placed upon theobject, the location of one or more of which is now unknown.

11—When the objects to be tracked are uniformed athletes such as icehockey players versus non-uniformed human subjects, their body sizes andshapes become less distinguishable due to the standard pad sizes oftheir equipment and their loose-fitting jerseys. As body shapes becomeless distinguishable, then the unique “constellation” of markers used toidentify a given uniformed athlete becomes less distinguishable and moremarkers must be added in order to clearly identify individual players.

The present inventors have addressed many of these drawbacks in theirco-pending applications entitled:

-   -   Multiple Object Tracking System, application Ser. No.        09/197,219; Filed: Nov. 20, 1998    -   Method for Representing Real-Time Motion over the Internet,        application Ser. No. 09/510,922; Filed: Feb. 22, 2000    -   Employing Electromagnetic By-Product Radiation for Object        Tracking, application Ser. No. 09/881,430; Filed: Jun. 14, 2001    -   Visibly Transparent Wide Observation Angle Retroreflective        Materials, application Ser. No. 09/911,043; Filed: Jul. 23, 2001

Each of these patent applications is hereby incorporated by referenceinto the present application.

In these patents applications, the present inventors describe variousaspects of a multiple object tracking system that functions in generalto track many types of objects but that is especially constructed totrack athletes during a live sporting event such as an ice hockey game.These patent applications teach at least the following novel components:

1—The system employs a matrix of separate overhead tracking camerasresponsible for first locating any given object as a whole in a local(X, Y) area rather than in a (X, Y, Z) volume coordinate system. Thistechnique yields a substantially uniform pixel resolution per areatracked providing a simple and regular approach to camera arrangementwhen the system is scaled to track larger areas.

2—The system employs separate sets of one or more pan, tilt, and zoomcameras per player to be tracked. These moveable cameras areautomatically directed by the system based upon the (X, Y) locationinformation that was first determined using the overhead tracking “area”cameras. Each of these volume cameras will collect (X, Y, Z) informationfrom a particular view of the player to be combined with at least the(X, Y) information captured by the “area” cameras concerning the sameplayer. Due to the system's ability to move and zoom eachplayer-tracking camera, a substantially uniform pixel resolution perplayer is achieved. This technique provides a simple and regularapproach to camera arrangement when the system is scaled to track moreand more players.

3—The system preferably employs a non-visible tracking energy such asultraviolet or infrared that is currently being generated bypre-existing lighting within the tracking area.

4—By using pre-existing lighting that is already in place with a purposeof illuminating the playing area for human observers, the system ensuresthat the observers will have a visible light indicator that the lamp ison. This will naturally keep the observer from staring at the lights andreceiving an overexposure of non-visible tracking energy.

5—The system employs one or more reflective, retroreflective,fluorescent, and fluorescent retroreflective materials that arespecifically designed to reflect only the chosen non-visible ultravioletor infrared tracking energy and to be substantially transparent tovisible light.

6—The system preferably employs markers that are made of ink which hasminimal thickness and can be placed upon virtually any surface such asin the case of hockey a player's helmet, jersey, or gloves; the tapethey use to wrap their stick; or the puck.

7—The system preferably encodes the player's unique identity into themarkings placed exclusively upon the “top surface” of the player, suchas the helmet or shoulders. In so doing, the player's identity can bedetermined solely from the (X, Y) area tracking cameras and issubstantially unaffected by player “bunching” and subsequent body markerinclusions that primarily affect the view of the body below the helmetand shoulders.

8—The system preferably takes advantage of the reduced player movementand smaller area of the playing surface entrance and exit as well as theteam benches in order to perform player identification. The uniquecharacteristics of the entrance and exit and benches provides theopportunity to focus the overhead (X, Y) tracking cameras in aconsiderably smaller field-of-view such that the players' helmets andattached markings are considerably enlarged with respect to the entirecaptured image. This in turn ensures that the space available for amarking on the helmet is sufficient to completely encode and thereforeidentify a given player through the use of more complex symbol patternssimilar to bar codes. As previously mentioned, since the unique playercode is therefore fully contained on the helmet, only the overhead (X,Y) cameras are necessary to determine identity thereby eliminating theeffect of body marker inclusions caused when the (X, Y, Z) cameras'fields-of-view are blocked.

9—By separating entrance and exit and bench tracking and identificationfrom playing surface tracking, it is possible to place a multi-frequencyresponsive marker at least on the player's helmet. For instance, thecomplex symbol patterns used to encode the player's identity can becreated with an UV ink while the helmet tracking mark can be createdwith an IR ink, or vice versa. This switching of frequencies effectivelydoubles the available marking area of at least the helmet andpotentially any other “top surface” such as the shoulders.

10—The area cameras have mutually exclusive fields-of-view with slightedge-to-edge overlap for calibration purposes. This calibration processis performed prior to live tracking.

11—The volume cameras are first calibrated with respect to their pan,tilt, and zoom drive mechanisms also prior to actual tracking. Theirfield-of-view will constantly overlap one or more area cameras. Thecombination of this overlapping area and volume information is then usedby the system for dynamic re-calibration and adjustment of the volumecameras. The system thereby permits individual cameras to move and berecalibrated simultaneously with actual tracking.

12—The system employs absorber compounds that are to be placed upon theobjects and playing surface prior to placing the markers in order to cutdown or eliminate unwanted reflections that are system noise.

13—The system employs predictive techniques based upon the object's lastknown position, acceleration, velocity, and direction of travel tominimize the search time required to locate the object in subsequentvideo frames.

Also currently existing in the commercial marketplace are the followingimportant components:

1—Wide-angle retroreflectors are capable of reflecting light in a widercone than typical retroreflectors. These are available in both cubecornered and microscopic bead optical body formats. They provide theopportunity to move the lighting source further away from the trackingcameras when using retroreflective materials.

2—Fluorescent laser dyes are capable of absorbing visible light justbelow the 700 nm wavelengths that are still visible and converting itinto IR light just above the 700 nm region that is non-visible. Whenusing IR, these dyes provide the opportunity to convert visible energyas emitted by pre-existing arena lighting into IR tracking energy thuseliminating the need to add lighting that specifically radiates IR intothe tracking volume.

3—Fluorescent laser dyes are capable of absorbing UV light around 330 nmwavelengths and converting it into UV light around 390 nm. Thisconversion is important given that certain commercially availablelow-cost digital imaging cameras are designed to have a higherresponsivity to UV light especially around the 390 nm range.Furthermore, existing arena lighting such as Metal Halide Lampscurrently generate UV energy in the frequency range of 315 to 400 nm. Byabsorbing the shorter wavelength UV energy around 330 nm and thenradiating additional UV energy around 390 nm, the fluorescent dyes willessentially “double up” on the preferred narrow band of trackingfrequencies.

4—Notch filters may be used with the tracking cameras and are capable ofpassing very narrow bands of specific frequencies of energy. Thisprovides the opportunity to place reflective, fluorescent, orretroreflective materials that operate at different frequency rangesonto different players to assist in the identification process.

The present inventors have described in their co-pending applicationsmany useful component and system solutions to the problems that areinherent within the existing systems. Additional components as describedabove also exist. It is possible to create several different and yeteffective machine vision systems for tracking multiple moving objectsbased upon the novel components disclosed within the present applicationand four co-pending applications. What is needed is an understanding ofhow all of these teachings can be combined to form several differentmachine vision systems, each with their own novel optimizations.

In addition to the aforementioned machine vision system solutions tomulti-object tracking it should be noted that at least two othercompanies are attempting to provide systems for similar purposes. BothTrackus, a Massachusetts-based company, and Orad, an Israeli-basedcompany, are attempting to develop real-time “beacon”—based trackingtechnology for sporting events. Orad has produced a working system tofollow horse racing; Trakus is the only company currently attempting tofollow players in an ice hockey game. While Orad's solution isessentially similar at the highest levels, the technology will beexplained based upon information gathered concerning Trakus.

Trakus' solution includes a microwave based transmitter and receiverthat will track a single point within the helmet of each player. Thereare many deficiencies with this proposed solution as compared to machinevision systems in general and the novel teachings of the presentinventors in particular. One of the most important distinctions is the“active” and potential harmful nature of the microwave technology. Ifused for tracking youth sports, it is anticipated that the averageparent would balk at the idea of placing even a low-power microwavedevice into the helmet of their child. Furthermore, there aresignificant reflection problems due to the hard interior surfaces of ahockey arena that must be resolved before this technology caneffectively track even a single point (the helmet) on every player onboth teams. As already discussed, machine vision-based systems employ“passive” markers that are capable of tracking 14 or more points (thehead and every major joint) on every player in real time. The presentinvention furthermore uniquely teaches a system that can also track gameequipment and the puck, devices that have surfaces that cannot besubstantially altered by the normal size of traditional markers.

SUMMARY OF THE INVENTION

For the purposes of disclosing the novel teachings of the presentinvention, the exemplary application of following the motion of hockeyplayers, their equipment, and the game puck in a live sporting eventwill be used to represent multi-object tracking.

In order to create an optimal multi-object tracking system, the presentinventors have focused on four major factors as follows:

A. The desired tracking information to be determined by the system;

B. The characteristics of the objects to be tracked;

C. The characteristics of the tracking environment; as well as

D. Traditional engineering goals.

A. With respect to the desired tracking information to be determined bythe system, the following characteristics were considered:

1. Is the desired representation to be visual for display only or amathematical model for measurement and rendering?;

2. Is two or three-dimensional information preferred?;

3. Is object orientation required in addition to location, velocity, andacceleration?;

4. Must this information be collected and available in real time?

5. What is the acceptable accuracy and precision with respect to thisinformation?;

6. Will the system be required to identify the objects as well astracking them?; and

7. Once an object has been identified, can this identity be lost duringtracking?

B. With respect to the objects to be tracked, the followingcharacteristics were considered:

1. Is the object rigid or flexible?;

2. In how many degrees of freedom will the object be moving?;

3. How fast will the objects be moving?;

4. Are there multiple objects to be tracked and, if so, how will thisimpact the tracking method?;

5. Are there any restrictions or safety considerations regarding thetype of electromagnetic energy used to track the objects?;

6. What are the physical space limitations on the objects for anymarker- or beacon-based tracking system?; and

7. What is the natural reflectivity of the various background surfacesto the different potential tracking energies?

C. With respect to the tracking environment, the followingcharacteristics were considered:

1. Is the setting “live” or “controlled”?;

2. What are the existing ambient electromagnetic energies?;

3. Are there any other pre-existing energy sources that may haveavailable by-product energy that could be used for tracking?;

4. What is the natural reflectivity of the various background surfacesto the different potential tracking energies?;

5. What is the size of the tracking area relative to the range of thepotential tracking methods?; and

6. Is the tracking environment physically enclosed within a building oroutside?

D. With respect to traditional engineering goals, the followingcharacteristics were considered:

1. The system should be scalable and therefore comprise uniformassemblies that can be combined into a matrix designed to increasetracking coverage in terms of area, volume, and the number of objectswhile still maintaining uniform performance.

2. The system should be minimally intrusive upon the objects to betracked and upon the surrounding environment, especially if thatenvironment is a live setting;

3. The tracking signal-to-noise ratio should be maximized; and

4. Manufacturing and installation costs should be minimized and theresultant system should be simple for the user to maintain and operate.

An optimized system design such as disclosed in the present applicationmust consider many of the above-stated questions and criteriasimultaneously. However, for the sake of consistency, each of thepertinent questions and criteria listed above will be considered inorder.

Therefore, referring first to the characteristics of the desiredtracking information to be determined by the system, the following istaught.

A.1. Is the desired representation to be visual for display only or amathematical model for measurement and rendering?

It is preferable that the tracking system creates a mathematical modelof the tracked players and equipment as opposed to a visualrepresentation that is essentially similar to a traditional filming andbroadcast system. A mathematical model allows for the measurement of theathletic competition while also providing the basis for a graphicalrendering of the captured movement for visual inspection from anydesired viewpoint. Certain systems exist in the marketplace that attemptto film sporting contests for multiple viewpoints after which a computersystem may be used to rotate through the various overlapping viewsgiving a limited ability to see the contest from any perspective. All ofthe aforementioned machine vision companies, such as Motion Analysis andVicon, generate a mathematical model.

This requirement of creating a mathematical model of human movementnecessarily dictates that at least one precise location on the humanbody be identified and followed. For beacon-based systems such as Trakusor Orad, following the beacon's signal provides both identity and objectlocation. To be implemented in a machine vision system, this furtherimplies that each of these locations remain substantially in view of twoor more tracking cameras at all times. Furthermore, the use of markersstrategically placed upon the players can greatly simplify video frameanalysis, as the markers become consistent center points that reduce theneed for detailed edge detection techniques as well as identificationand weighted averaging of player surfaces. This marker technique isimplemented by all of the vision-based systems such as Motion Analysisand Vicon and is preferred by the present inventors. As will bediscussed, the exact choice of the type, shape, size, and placement ofthe preferred markers is significant to some of the novel functions ofthe present invention and is different from existing techniques.

A.2. Is two or three-dimensional information preferred?

Three-dimensional information provides the ability to generate morerealistic graphical renditions and to create more detailed statisticsand analyses concerning game play. All of the aforementioned machinevision companies such as Motion Analysis and Vicon attempt to generatethree-dimensional data while the beacon-based Trakus and Orad onlygenerate two-dimensional information.

This requirement of creating a three-dimensional mathematical model ofgame play further dictates that at least the major joints on a playerare identified and tracked. For instance, the player's helmet,shoulders, elbows, writs, torso, waist, knees, and feet are allbeneficial tracking points for a 3D model. This informational goal inpractice precludes beacon-based systems such as Trakus and Orad since itwould require a significant number of beacons to be placed onto eachplayer, often times in locations that are not convenient. Furthermore,each beacon will create additional signal processing and, given currentstate of the art in microwave tracking, the system could not functionquickly enough to resolve all of the incoming signals.

For vision-based systems such as Motion Analysis and Vicon, significantdifficulties also begin to present themselves in consideration of therequirement that each joint be in view of at least two cameras at alltimes. As players move about and change their body positions, individualjoints can easily be lost from view (inclusion) or take positions thatfrom a given camera's “flat 2D” perspective appear to make them a partof a different player. This is especially true in light of the smallspherical markers used by existing systems that are not in view fromevery possible rotational angle of the joint. What is needed is avision-based tracking system that can identify and track players withone set of cameras and then automatically direct a second set of camerasto adjust their views so as to minimize these inclusions. What isfurther needed is a tracking system that employs markers that can covera much larger area, for instance all the way around an elbow rather thana single point or set of three points upon the elbow, while at the sametime remaining less obtrusive.

A.3. Is object orientation required in addition to location, velocity,and acceleration?

Especially in the sport of ice hockey, it is important to know theorientation of the player as a whole since, for instance, they can beskating in a certain direction either facing forwards or backwards.Furthermore, in any motion system designed to follow the movement of ahuman joint that has multiple degrees of freedom, it is necessary todetermine the orientation of the joint, not merely its position in orderto create an accurate mathematical model. This requirement exceeds thecapacity of beacon-based systems such as Trakus and Orad since theemitted signals are uniformly omni-directional and therefore cannot beused to determine rotation about the transmitting axis. For vision-basedsystems such as Motion Analysis, Vicon and the present invention, thisrequirement significantly favors the use of markers over edge detectionof player surfaces. This is because the markers have clear center pointsand could be placed in a triangular format that is a traditional methodfor orientation detection. The present inventors prefer larger markingsthat form shapes such as an oval, circle, or triangle over the placementof spherical markers as is currently practiced.

A.4. Must this information be collected and available in real time?

The ability to capture and analyze images, convert them into a 3-Dmathematical model, and then dynamically render a graphic representationalong with quantified statistics in real time offers significantopportunities and challenges. For beacon-based systems such as Trakusand Orad, there is a single set of receiving towers throughout the arenathat must process as many beacons as required to follow all of theplayers to be tracked. Essentially, the entire set of towers isnecessary to cover the rink for even a single player. Additionally, eachplayer's beacon signal will be picked up by each tower and then must becompared across all towers to perform the location function. As moreplayers are added, they will create additional processing for eachtower. Hence, the total number of players is limited by the capacity ofa single tower to process its received signals in real time. While suchbeacon-based systems may be scalable by playing area, therefore simplyadding additional towers will cover additional area, they are notscalable by the number of players. Hence, for more players within agiven area, one cannot simply add more towers. These systems areinherently “player bound” as opposed to “area bound.”

For vision-based systems such as Motion Analysis and Vicon, the goal ofreal-time information significantly challenges system capacityespecially when combined with the need to track multiple players. Thisrequirement limits the amount of image processing time available tohandle ambiguities created in the data set by the inclusions that occurwhen players overlap, or bunch up, within a given camera view. Similarto the beacon-based systems, these camera systems can be said to bescalable by playing area since each additional playing volume requiresthe uniform addition of fixed perspective cameras. However, they too are“player bound” since any number of players may at a single time end upin any given playing volume creating a large, “included” data set thatcannot be sufficiently processed in real time if at all without humanassistance to clarify ambiguities.

The present inventors prefer to separate the function of “area” trackingfor 2D movements and identification from “player” tracking for the full3D data set. Hence, a matrix of overhead (X, Y) cameras follows themovement of any number of players per single camera across a singleplaying area. Since the number of markers tracked from the overhead viewis limited to primarily the helmet and shoulders, the total number ofmarks, even for a large number of players is still trackable.Furthermore, due to the overhead view and the tendency for players toremain upright, it is expected that there will be minimal instances ofhelmet or shoulder “overhead inclusions.” In addition to being trackedin 2D space by the overhead cameras, each player will also be followedby at least two and preferably four dedicated pan, tilt, and zoomperspective cameras. These dedicated, movable, perspective cameras willbe automatically directed by the known (X, Y) location of each player asdetermined by the overhead cameras. This combination of novel techniquesprovides for scalability by both tracking area and player. Hence, tocover more area simply add one or more overhead “area” tracking cameraswhile to cover more players simply add additional sets of one or morededicated movable “player” tracking cameras.

A.5. What is the acceptable accuracy and precision with respect to thisinformation?

The speed of a hockey puck can approach 100 mph while a player may beskating upwards of 25 mph. As was previously described in detail in thepresent inventors' co-pending application for a Multiple Object TrackingSystem, minimum capture rates of 40 frames per second for low-endcommercial video cameras will follow pucks at a maximum of 3.7 feet perframe and players at 0.9 feet per frame. While this is sufficient tocreate a realistic mathematical model of game movement, faster cameraswith between 2 to 4 times the capture rates are readily available. Theincorporation of faster cameras also requires the use of fastercomputers and software algorithms to locate and identify markers andtherefore players and equipment within the allotted fraction of asecond. Currently, the beacon-based Trakus captures data at 30 locationsper second.

A.6. Will the system be required to identify the objects as well astracking them?

The ability to identify the tracked objects is critical and poses a moredifficult problem for the video-based systems versus the beacon systems.With a beacon system, the transmitted signal can easily contain auniquely encoded value that serves to differentiate each player. For avideo-based system, which uses a single tracking energy and passivemarkers, the encoding must be accomplished via some unique arrangementof markings. For vision based systems such as Motion Analysis and Vicon,the current practice is to consider the unique “constellation” ofmarkers as placed upon each players body to form an encoding for thatplayer. This thinking is predicated on the idea that no two individualshave exactly the same body shape and hence the markers will always be inat least slightly different configurations. If two players did end upwith “constellations” that were too similar in configuration to beaccurately differentiated, then one or more additional markers would beadded to at least one of the players to sufficiently differentiate theone from the other. It should be noted, that with this strategy, themajority of the markers play a dual role of both joint and body segmenttracking on an individual basis and player identity on a collectivebasis.

A further practical limitation of this technique is its real-lifeimplementation with non-sophisticated system operators. In other words,at the most difficult youth sport level, the ideal tracking system mustbe operable by minimally trained lay people such as coaches and parents.Placing these markers upon the player's body in such a way as toguarantee a unique constellation for all 16 to 20 players on a team willbe overly restrictive. What is needed is a simple approach to placing aminimum of a single marker upon the player that is assured to provide aunique identification during game tracking.

The present inventors prefer to isolate player identification to thosemarkings that can be viewed strictly from the overhead cameras, i.e.,the “top surface” of the player's body. This minimizes the occasionswhen the markers that are relied upon for identification are hidden(included) from camera view based upon player bunching. In the preferredembodiment, the markings placed on the player's helmet and possibly alsoon their shoulders form a uniquely identifiable pattern, similar in formto bar coding. The decision as to whether the shoulder markers need tobe included in the unique pattern depends mainly upon the total areaprovided by the helmet alone and the number of pixels per inchresolution of the viewing cameras. For effective machine vision, anymark to be “seen” must be picked up by preferably two pixels. Dependingupon the pixel per inch resolution of the camera configuration, thesetwo pixels will represent a different sized area. In practice, due tothe expected movement and rotation of the helmet that is inherently nota flat surface, it is preferred to have many more pixels per inch ofmarking for accurate recognition. Additionally, it is anticipated thatthere may be between thirty to forty players and game officials on theice over the course of a given game. Each one of these players andofficials must have a unique tracking pattern on their helmet. Hence, ifthere is not enough room on the helmet to include an easilyrecognizable, unique encoding for each player, the shoulders can be usedto augment the coding scheme.

This novel approach provides the opportunity to optionally separate theplayer identification process from the tracking process. Since theplayers are all expected to enter and exit the playing area via alimited passageway and furthermore to collect just outside the playingarea on team benches, it is anticipated that these will present idealareas to first ascertain and then reconfirm unique player identities.Once the players are identified as they first enter the playing areathrough the passageway, they can then simply be tracked as they movethroughout the playing area. Should an ambiguity arise between twoplayers during game play, their identity can be sorted out once theyreturn to the team benches.

This separation of player identification from movement tracking opens upthe possibility of different camera and lens configurations in the“identification areas” versus the “tracking areas.” Hence, theidentification cameras could for instance be focused on a much narrowerfield of view since the passageway and team bench are considerablysmaller in area than the playing surface. By narrowing the field ofview, the resultant pixels per square inch of “id pattern” will beincreased thereby reducing the necessary size of this unique marking.Other possibilities are anticipated such as the creation of a uniquemarking that is detected in a first select frequency such as IR and anoverlapping tracking mark that is detected in a second select frequencysuch as UR. (These frequencies could easily be reversed with no changeto the novel functionality.) In this embodiment, the “identificationareas” will be viewed with IR cameras while the “tracking areas” will beviewed with UV cameras. It should be noted that both the IR id mark andthe UV tracking mark would be non-visible to the players and viewingaudience. Furthermore, these separate “identification areas” will alsofacilitate tracking the limited movement of the players until they enterthe playing surface, i.e. “tracking area.”

A.7. Once an object has been identified, can this identity be lostduring tracking?

The answer to this question is entirely driven by the selected trackingand identification technique. For instance, with beacon-basedidentification such as Trakus and Orad, the identity is only lost if theunique player signal is lost. For traditional vision-based tracking suchas Motion Analysis and Vicon, player identity can be easily lost basedupon a number of factors. These factors include the number of camerasviewing a given playing volume, the pixel per inch resolution for theallowed marker size as well as the number of expected inclusions due toplayer bunching. The final factor is randomly variable throughout agiven tracking session. The preferred embodiment of the presentinvention has separated the identification of each player into a 2Doverhead viewing function that strictly focuses on the “top surface” ofeach player. Since the number of inclusions of player's helmets isexpected to be minimal, especially from the overhead view is expected tobe minimal, the preferred embodiment will not experience the samedifficulties in maintaining player identity as traditional systems.

B. Referring next to the characteristics of the objects to be tracked,the preferred and alternate novel embodiments must consider at least thefollowing factors:

B.1. Is the object rigid or flexible?

The majority of objects (i.e., the players) are flexible while someobjects are rigid such as the sticks and puck. Flexible objects such asplayers create difficulties for vision-based tracking systems becausetheir form is constantly changing. These changes create greateropportunities for inclusions (blocked markers) and for misinterpretationwhen multiple players wearing multiple markers are each partially inview from any given camera. For beacon-based systems such as Trakus andOrad, the only location being tracked on the player is the helmet thatis itself rigid. Due to the nature of the beacon itself, it isimmaterial whether the object is flexible since the beacon's signal willtransmit through the player's body and most equipment. For machinevision systems such as Motion Analysis and Vicon, this restrictionsignificantly adds to the difficulty of keeping any given marker in theview of two or more cameras at a single instant, especially in light ofthe multiple player requirements.

Due in part to the infinite variations of marker images that can becreated by any number of players holding their bodies in any number ofpositions creating any number of inclusions, all within a single camerafield of view, the present inventors favor the novel teachings of thepresent invention. More specifically, by isolating player identificationto markings placed upon the player's helmet that is rigid andsubstantially in view of the overhead cameras at all times, theflexibility of the player's body is removed as a negative factor for themore complex purpose of identification. Additionally, the use of largerink markings provides for various marker shapes such as circles aroundthe elbows, torso, knees or ankles or strips that can run down aplayer's arms, legs, torso or stick, etc. This in turn allows thepresent invention to greatly reduce the chances for total inclusion asthe player's flex individually and bunch together. And finally, theseparation of overhead (X, Y) “area” tracking that automatically directsthe dedicated pan, tilt and zoom “player” tracking cameras, which inturn collect the 3D information, further minimizes the inclusionscreated by multiple flexible objects.

B.2. In how many degrees of freedom will the object be moving?

Similar to the difficulties created by the flexibility of a player, thefull six degrees of freedom within which a player and or their limbs maytravel creates increased challenges for vision-based solutions such asMotion Analysis and Vicon. It should first be noted that beacon-basedsystems such as Trakus and Orad are not impacted by the freedom of theplayer to move their helmet in any of the six possible directions fromthe current location. Of course, the beacon solution itself has a muchgreater shortcoming since it is in practice limited to a single beaconper player.

Returning to the vision systems such as Motion Analysis and Vicon, thecurrent practice is to place small spheres ranging in size fromapproximately ⅛″ to 1″ in diameter upon the various joints and locationsto be tracked on any given player. Often, these small spheres arethemselves held out away from the players body on a short stem ofapproximately ½″. This configuration is uniformly adopted by the current3D vision-based tracking systems and represents a significant systemlimitation. The purpose of the spherical shape of the marker is toensure that a maximum image size is created irrespective of the locationof the marker (and its associated body part) with respect to the viewingcamera. Hence, no matter what the angle of view, the marker will alwaysshow up as a circle unless the view is in some way blocked. The purposeof the stem is to hold the sphere out away from the body therebyreducing the circumstances of partial inclusion created by the bodysurface to which the marker has been attached. While this technique hasworked well for “controlled” situations, these markers are unacceptablefor “live” events.

What is needed is a method of marking the player's joints that will besubstantially visible from two or more cameras from any view pointdespite the six degree freedom of player movement. Given thisrequirement, the present inventors prefer the novel technique of using amuch larger “surface area” marking. This essentially maximizes themarker view by both increasing the size of the mark as well ascontinuously surrounding a joint or body part such as the elbows, torso,knees or ankles. By both enlarging the mark and surrounding the bodypart, the ability for that part to been seen from any angle while itmoves in six degrees of freedom is significantly increased.

B.3. How fast will the objects be moving?

As was previously stated, the speed of a hockey puck can approach 100mph while a player may be skating upwards of 25 mph. When players arespinning or turning at or near their full speed of travel, the combinedjoint movement speed can present problems to tracking systems with lowersampling rates. For beacon-based systems such as Trakus and Orad, whichtake approximately 30 locations per second, these speeds would present asignificant challenge if the beacons could be placed upon a puck or evena player's wrist. For vision-based systems such as Motion Analysis,Vicon and the present invention, camera and video capture technology canpresently handle upwards of 240 frames per second with acceptable 1megapixel resolution.

B.4. Are there multiple objects to be tracked and, if so, how will thisimpact the tracking method?

As has been previously suggested, it is necessary to track the motion ofat least 10 players and 3 game officials in the playing area for thesport of ice hockey. Including the players' benches, the total number ofplayers and officials can exceed 40. For other sports such as Americanfootball, the total players and officials can exceed 60, over 20 at atime on the playing surface. This requirement for ice hockey alone haspushed the limits of beacon-based systems such as Trakus and Orad thatare simply attempting to track a single point, the helmet, for eachplayer. For vision-based systems such as Motion Analysis and Vicon, thisrequirement also presents a significant challenge. All of the existingvision tracking systems first place fixed field of view cameras justoutside the tracking area. These cameras are strategically placed in anattempt to keep the maximum number of markers within view of at leasttwo cameras at all times even as these markers travel about in sixdegrees of freedom. The prospect of viewing multiple players within agiven volume further challenges the strategic placement of these fixedcameras. As more and more players randomly congregate within any givenvolume, more and more of their respective markers will be blocked, fromany given camera's field of view. Since the sum total of a player'smarkers is used in and of itself for player identification, as markersare lost from view this not only jeopardizes the accurate tracking of anindividual body part, but also increases the incidence of improperlyidentified players.

The present inventors prefer the novel approach of controllablydirecting one or more movable pan, tilt and zoom cameras to follow eachindividual player based upon the first determined (X, Y) location asobtained via the overhead cameras. In this way the system's “totalfield-of-view” is dynamically recreated and maximized for player andmarker visibility. The preferred embodiment includes movable camerasthat are not necessarily unconditionally dedicated to follow a singleplayer. It is anticipated that, given the known and projected locationsof each player in the tracking area, it may be beneficial to dynamicallyswitch one or more cameras away from one or more players onto one ormore other players.

B.5. Are there any restrictions or safety considerations regarding thetype of electromagnetic energy used to track the objects?

For safety requirements the properties of both wavelength and flux mustbe considered apart and in combination. The beacon-based trackingimplemented by Trakus employs microwaves similar to cell phonetechnology. These frequencies of energy can be harmful in larger amountsand, for this reason, Trakus pulses their transmitter signals so as toreduce the average energy exposed to the player through the helmet.However, the present inventors see at least the potential for perceivedharm since the players, which in the case of recreational hockey areyouths, will be exposed to higher dosages of this energy as they tend tobunch together during play or at least in the team bench area.

In the case of vision-based systems such as Motion Analysis or Vicon,the primary tracking energy is visible, such as red light. While theseenergies are not harmful to the players, they do restrict the use ofthese systems to controlled settings where it is acceptable for theathletes and viewers to see the markers and their reflections. MotionAnalysis, Vicon and other providers also offer the choice of working innear IR frequencies. These IR frequencies are safe for human exposure.The ring lights used to illuminate the tracking area in these systemsonly emit IR energy, however, and as such are not noticeably active onto the casual viewer. There is some concern that an uninformedindividual could stare up at the camera with its IR ring light andreceive an overexposure of IR energy possibly damaging the retina of theeye. For this reasons, these ring lights come with additional smallLED's that emit a visible light cue when the IR light is on, essentiallyalerting people not to stare. This may not be a sufficient mechanism forthe uniformed casual viewer. When tracking in IR energy, the presentinventors prefer a light source that emits both visible and IR lightsince the visible light will act as a natural cue to dissuade theobserver from staring.

Also taught in the preferred embodiment is the novel use of UV energyfor tracking purposes. UV energy is broken into three types; UVC, UVBand UVA. UVC and UVB are generally thought to be the most damaging whileUVA is considered to be biologically safe and much more prevalent in theatmosphere than UVB. UVA is typically considered to be those wavelengthsbetween 315 to 400 nm. Visible light begins as 400 nm. The preferredembodiment employs tracking frequencies centering around 390 nm, justshy of visible light. This is an ideal wavelength since there arelow-cost industrial digital cameras whose responsivity curves peak at ornear this frequency. Furthermore, the metal halide lamps that are oftenfound in hockey rinks generate a significant amount of UVA energy thatcould be used as an energy source.

B.6. What are the physical space limitations on the objects for anymarker or beacon-based tracking system?

To employ beacon-based tracking such as Trakus and Orad for contactsports it is preferable to embed the beacon somewhere in the player'sequipment. Due to several other drawbacks, these beacon systems havebeen limited to tracking a single point that was centrally chosen to bethe helmet. If other considerations were permitting and additionalbeacons could be tracked per player, then the requirement to embedbeacons into the equipment in order to avoid injury possible from playercontact would itself becoming a significant problem.

For vision-based systems such as Motion Analysis and Vicon, the markersmust reside outside of the player's exposed surface in order to receiveand reflect the tracking energy. Furthermore, as was previouslymentioned, these markers have been constructed as spheres so as tomaximize the reflected image independent of marker angle to camera.These spheres strategically placed onto various joints and body partsare impractical for live, uncontrolled settings such as a full-contactsporting event. The preferred embodiment of the present inventionspecifies the use of reflective ink or paint that can be applied to thevarious substrates such as plastic, wood, fabric, leather, rubber, etc.and will add minimal thickness. Unlike to current practice of usingvisible spheres of limited diameter that must protrude from the playerto maximize reflection, the preferred technique uses non-visible ink toplace large markings on the various joints and body parts. Dependingupon the transmissivity to the fabric being worn by the players, the inkitself could be placed on the inside of this material and receive andreflect the non-visible tracking energy directly through the jersey. Theend result of this type of preferred marking is to overcome any physicalspace limitation considerations.

B.7. What is the natural reflectivity of the various background surfacesto the different potential tracking energies?

The reflectivity of the background surfaces, and for that matter the“non-tracked” foreground surfaces, represents the most significantsource of system noise. For beacon-based systems such as Trakus andOrad, the microwaves emitted by the players' transmitters will reflectoff hard surfaces such as metal and concrete. Due especially to theenclosed environment of a hockey arena, these reflections will continueto bounce between the background surfaces within the tracking volume asthey slowly attenuate, thus contributing to significant noise problems.

For vision-based systems such as Motion Analysis and Vicon, reflectivityissues are limited to the individual camera's field of view. Hence,since the cameras are primarily focused on the playing surface andsurrounding boundaries, the reflection issues are already diminishedover beacon-based systems. However, for the sport of ice hockey, theplaying surface is ice that tends to be highly reflective of visible, IRand UV energies. In the case of visible red light as preferred by MotionAnalysis and Vicon, these reflections are typically handled via softwareprocessing. During system calibration, the images captured by eachcamera are reviewed for possible ice surface reflections especially fromthe ring lights affixed to the tracking cameras. Any such undesirablesignals are “mapped” out via a software tool that essentially informsthe image analysis to ignore any and all data capture from thosecoordinates. Of course, should valid marker data coincidentally show upat that same location during a live filming, it would be ignored aswell.

In the preferred embodiment, absorbent compounds are applied to thevarious background and “non-tracked” foreground surfaces. For instance,if the tracking energy is UV, traditional UV absorptive compounds as arewell known in the commercial marketplace can be used to absorb stray UVenergy. The entire player including their jersey, helmet, pads, gloves,stick, etc. as well as the ice surface, boards and glass can all befirst treated with one or more UV absorptive compounds. This noveltechnique significantly reduces system noise and essentially makeseverything but the subsequently applied markers invisible to thetracking cameras. Similar techniques are possible if IR is used as thetracking energy.

C. Referring next to the characteristics of the tracking environment,the preferred and alternate novel embodiments must consider thefollowing questions:

C.1. Is the setting “live” or “controlled”?

Tracking players in a live sporting event such as ice hockey adds amajor restriction to the system requirements. Namely, the system must beas unobtrusive as possible for both the players and the viewingaudience. In the case of the beacon systems from Trakus and Orad, thetransmitters are hidden and the tracking signal is non-visible. The onlypossible drawback is that receiving towers must be placed throughout thearena. Careful attention must be made to keep these towers from blockingthe view of nearby fans.

With respect to vision-based systems such as Motion Analysis and Vicon,there are two major types of restrictions present in their approach.First, the tracking energy itself is typically within the visiblespectrum (e.g. red light). This energy is necessarily also visible tothe players and audience. Second, as previously discussed, the markershave substantial size. Their size makes them noticeable to players andfans and potentially dangerous to the players since they could applyadditional pressure to their bodies upon contact with other players, theice surface or the boards. Furthermore, these markers could easily bedislodged creating at a minimum the stoppage of game play and or worse apotential injury hazard. Motion Analysis and Vicon, as well as othervision tracking providers, offer an option to work in IR light. Thischange eliminates the direct interference of visible light and itsreflections off the ice surface with player and fan vision.

However, the markers used by these types of systems have an additionalproblem other than their size. Specifically, they are constructed of aspherical material preferably a foam ball, that has been covered with aretroreflective tape. The tape itself is greyish in visible color. Thesetapes are typically made of a material referred to as “cube corneredretroreflectors” but could also be produced with microspheric devices.In either case, current technology provides for optically transparentbodies that are coated on their undersides with a reflective materialsuch as aluminum or silver, essentially forming a mirror. The additionalproblem is the broad band frequency response of these microscopicmirrors that includes visible light as well as UV and IR. Hence, even ifthe tracking energy is switched to IR, other ambient visible frequencieswill themselves retroreflect off these markers back towards the playersand fans who are near the light source. As players move about, thesespheres will have a tendency to fluctuate in brightness as they passthrough various lighting channels. If a fan or reporter were to useflash-based photography or illuminated videotaping, these markers wouldretroreflect this energy back into their camera causing noticeablebright spots on top of the greyish circles.

To overcome these drawbacks, the present inventors teach the use ofink-based markings that are applied directly to the substrates worn orheld by the players. Unlike the spherical markers, these inks will notadd any appreciable thickness to the player's clothing or equipment.This effectively overcomes the marker size problem. To overcome thebroad band reflectivity drawback, the preferred embodiment includeseither reflective, fluorescent or retroreflective inks and compoundsthat have been engineered to only reflect the narrow band of non-visibletracking energy and as such are substantially transparent to visiblelight.

C.2. What are the existing ambient electromagnetic energies?

Within the live setting of at least a hockey arena, there will be largearea high intensity discharge (HID) lighting such as Metal Halide orMercury Vapor lamps. These sources typically are chosen because theygenerate a very broad range of energy throughout the visible spectrumproducing a natural-looking white light. The placement of these lampswill be determined and conducive to audience viewing considerationsrather than object tracking requirements. During the contest, theselights may be altered in intensity or augmented with additional lightingfor visual effect. There is no assurance that the illumination levels inthe arena will remain consistent in intensity. Furthermore, it must beanticipated that either fans or broadcasters will bring in additionalportable light sources for filming purposes that will also add to theuncontrolled lighting levels.

For beacon-based systems such as Trakus and Orad that depend uponmicrowave transmissions, the visible light frequencies do not presentany noise to the tracking system. Fans using portable cell phones willbe transmitting microwave energy but the beacon signals can simply beset to emit at a different wavelength.

For vision-based systems such as Motion Analysis and Vicon, theseambient frequencies are an important source of noise especially inconsideration of the various colored uniforms and equipment worn byplayers during a typical live contest. Hence, if the players are wearingred uniforms or they have red streaks on their jerseys or equipment,then the ambient visible light sources will tend to reflect this colorinto the tracking cameras. These reflections are similar to those causedby the red LED ring lights as their emitted energy strikes the icesurface. As previously mentioned, the reflections from the ring lightsare “mapped out” via software during system calibration. This softwaretechnique could not be used to “map out” the dynamically changing redlight reflections caused by the player's jersey and equipment colors. Itshould be noted that these systems include red light filters on theircameras effectively restricting noise to a narrow band overlapping theemitted tracking energies, but not fully eliminating the problem.

As previously mentioned, machine vision companies such as MotionAnalysis and Vicon do offer their systems using IR light. Thisnon-visible frequency is used when the customer requires that thetracking be done in darkness. The IR tracking energy is not being usedto reduce system noise by cutting down on the visible spectrum.

In order to avoid noise created by the existing visible light sources,the present inventors prefer to work in a non-visible tracking energysuch as UV or IR and to employ energy-absorptive techniques elsewheredescribed in the present and co-pending applications.

C.3. Are there any other pre-existing energy sources that may haveavailable by-product energy that could be used for tracking?

Several of the HID lamps used for large area illumination also generatenon-visible frequencies such as UV and IR. Typically, the manufacturersof lights encase the inner bulb and filament with an outer bulb that hasbeen filled with a special vapor mixture and coated with a specialcompound. The combination of the additional glass bulb, the specialvapor and the special coating act to absorb the non-visible energiesrather than release them into the surrounding environment. The presentinventors anticipate modifying these various light sources tospecifically emit some additional portion of their generated non-visibleenergy that is currently being absorbed.

C.4. What is the natural reflectivity of the various background surfacesto the different potential tracking energies?

Within a hockey arena, the hard mirror-like ice surface presents achallenge to both beacon- and machine vision-based tracking. Forinstance, other playing surfaces such as grass would tend to absorb mostof the ambient frequencies including microwave, red light, UV and IRwhile only reflecting green light. Another reason the present inventorsprefer to track the players and equipment using a non-visible energysuch as UV or IR is that the background can then be first treated witheither non-visible absorbers, reflectors, or both. This helps to createa clear reflective distinction between the markers and the backgroundthereby facilitating image analysis. The present inventors also preferto place these non-visible absorbers or reflectors onto the foregroundobjects such as the players and equipment prior to marking them. In thisway, the detectable intensity levels of any unwanted reflections off theforeground objects can be controllably differentiated from the marker'sreflections.

C.5. What is the size of the tracking area relative to the range of thepotential tracking methods?

Some sports have relatively small playing areas and few players, such astennis, while other sports have very large areas and many players, suchas golf. Sports such as ice hockey, football, soccer and baseball havemid-to-large sized areas with many players to track simultaneously.Microwaves such as those used in the beacon-based systems from Trakusand Orad work well in any of these various-sized playing surfaces.Vision-based systems as currently implemented by companies such asMotion Analysis and Vicon are challenged to track areas significantlylarger than a tennis court in size.

The major reason for their difficulties is the “volume tracking”approach they have taken to solving this problem. Simply put, eachcontestant may move about the playing surface which because of his orher own height and the expected flight of any game objects, becomes aplaying volume rather than an area. Every portion of this volume must bein view of two or more cameras at all times. The cameras must be fixedprior to the contest so that they may be calibrated as a network. Ifplayers will have a tendency to bunch up, additional cameras will beneeded within any given volume to create additional views therebyreducing anticipated inclusions. The cameras must be limited in thefield of view so that they can maintain a sufficient resolution orpixels per inch within their field in order to detect the reflectedmarkers. As the playing area widens, it becomes increasingly difficultto place cameras close enough to the inner volumes so that the idealfield of view is maintained per camera without causing an obstruction tothe players or viewing audience. This obstruction would occur if amounting structure were created to hang the cameras directly above theinner volumes. Newer cameras will continue to provide higher resolutionstheoretically allowing the cameras to move further back from any givenvolume and still maintain the requisite pixels per inch resolution. Ascameras pull back, however, the distance between the energy trackingsource, the reflective marker and the cameras will continue to increasethereby having a negative effect on signal strength.

The present inventors prefer to separate player (object) tracking intotwo distinct sub-processes thereby eliminating the aforementionedproblems. In distinct contrast to the “volume tracking” approach, thepreferred embodiment of the present invention relies upon a “playerfollowing” controlled by an “area tracking” technique. In essence, theplayers are first tracked in two dimensions, X and Y, throughout theplaying area. The currently determined location of each player is thenused to automatically direct that player's individual set of camerasthat “follow” him or her about the playing surface. This two-stepapproach has many critical advantages when faced with tracking objectsthroughout a larger volume. First, locating the “top surface” (i.e.,helmet) of each player in X-Y space for substantially the entire contestis significantly simpler than trying to detect their entire form fromtwo or more cameras throughout the entire playing volume. Second, byplacing the player id on their “top surface” (i.e., their helmet and ifneed be shoulder pads), the system is able to easily identify eachplayer while it also tracks their X-Y coordinates. Third, bycontrollably directing one or more automatic pan, tilt and zoom camerasto follow each player (and game object) the ideal field of view andmaximum resolution can be dynamically maintained per player.

C.6. Is the tracking environment physically enclosed within a buildingor outside?

The indoor enclosure is most prevalent in the sport of ice hockey asopposed to other major sports such as football, soccer and baseball.This particular requirement has a negative effect on beacon-basedsystems such as Trakus and Orad since the hard surfaces of the enclosedarena will cause the microwave tracking energy to go through manyreflections before finally dissipating. For vision systems such asMotion Analysis and Vicon, the lower ceilings that are typically foundin local youth hockey arenas present a different type of problem. Due tothe “volume tracking” approach just discussed, these systems ideallyrequire the placement of their tracking camera assemblies above theplaying surface or at least off to the side and very near the “trackingvolume.” These assemblies include the camera, lens and filter, the ringlights and power supply as well as a small computer processor forinitial data analysis and may typically cost around $15,000. In totaldistance, each assembly may not be more than twenty to thirty feet offthe ice. This makes each assembly prone to damage when a puck is eitheraccidentally or intentionally shot at the camera system. The presentinventors prefer an enclosed assembly where the camera, lens and relatedequipment are protected and yet still able to view the ice surface belowthrough a Plexiglas or similarly transparent covering.

D. Referring next to traditional engineering goals, the preferred andalternate novel embodiments must consider at least the followingfactors:

D.1. The system should be scalable and therefore comprise uniformassemblies that are combinable into a matrix designed to increasetracking coverage in terms of area, volume or the number of players(objects) while still maintaining uniform performance.

In one respect, beacon-based systems such as Trakus and Orad include auniform assembly in the form of the player's transmitter that isembedded within their helmet. This design is scalable by the number ofplayers since each additional player is simply given an additionaltransmitter. In order to be scalable in terms of the subsequent signalprocessing required to locate each transmitter in X-Y space, however,the system will need to encode select transmitters to select receivingtowers. The present inventors do not believe that companies such asTrakus and Orad are currently practicing this technique. This conceptrequires that a capacity be determined for a single set of receivingtowers in terms of the number of transmitters that can be tracked inreal time. For example, assume that four receiving towers are initiallyset up within the arena and that they can only adequately track fivetransmitters in real time. Once a sixth player is outfitted with atransmitter, the entire system will lose its real time processingcapability. It is preferable to equip the initial four towers with afirst processor that eliminates any received signal that was notgenerated by one of the initial five transmitters. In this way theinitial five players remain adequately tracked in real time. In order totrack an additional five players, an additional four towers andprocessor should be added to the arena. These towers would then beassigned to specifically process only those transmitters with the uniquecodes corresponding to the second five players. This approach will makethe beacon-based system scalable. Otherwise, the entire system will be“player bound” in that it will have a maximum number of totaltransmitters that can be tracked irrespective of the playing volume.

Similarly, instead of encoding each signal and digitally filtering allincoming signals for selected codes, each maximum group of transmitterscould be assigned a different frequency. The filtering process couldthen be moved to electronics. Another possibility is to coordinate thetiming of the transmitted signals from the various groups of maximumtransmitters. In essence, each group of transmitters will bebroadcasting their locating signal as specified and non-overlappingintervals coordinated with the receiving towers. The important point isto make the system scalable by defining a minimum configuration oftracking apparatus that can simply be repeated to either increase area,or in the case of the beacon approach, increase players tracked.

For vision-based tracking systems such as Motion Analysis and Vicon, thesystems are not uniform in their “camera view per volume.” This lack ofuniformity works against system scalability and is due to severalcoincident design factors. First, the camera assemblies are bothrelatively expensive in their own right and when taken togetherrepresent a significant portion of the system price. An attempt is beingmade to maximize the use of each individual camera's field of view.Second, the field of view of each individual camera is best thought ofas a four-sided pyramid where the apex emanates from the CCD array inthe back of the camera. In practice, the first several feet of the fieldof view are not useable for tracking and even so are more than likelyoutside of the tracking volume for pragmatic clearance considerations.This can be thought of as taking the top off the pyramid. The pixels permarker inch resolution at the nearest versus farthest points in thispyramid can be substantially different and must necessarily be limitedby the minimum acceptable resolution to identify a given size marker.

Third, to be scalable, each tracking volume serviced by two or moreassemblies should be either square or rectangular in its cross-sectionalshape parallel to the ice surface. In this fashion, these volumes couldsimply be repeated with slight overlaps for calibration purposes inorder to create larger and larger tracking volumes. However, this isdifficult to accomplish in practice using overlapping, four-sidedpyramids. And finally, an individual player with multiple markersattached could transverse any portion of the tracking volume at anyrotational angle and it is necessary that each marker remain in the siteof two cameras at all times. This requirement calls for a minimum offour cameras surrounding any given volume. In practice six cameras is amore acceptable minimum. As additional players are added to the trackingvolume, it is not difficult to see how easy it will be for one player toblock the camera's view of another. Keeping in mind the playeridentification technique practiced by companies such as Motion Analysisand Vicon where the unique “full body” constellation of markersidentifies a skater, inclusions take on greater significance.

The task of designing a uniform configuration of acceptable resolutionthroughout the entire tracking volume while also accounting for the twocamera minimum per marker and multiple player bunching is formidable.Furthermore, the end result “wastes” camera field of view since theactual cross-sectional shape formed by the configuration will not beeither a square or rectangle nor will it even be uniform in size as itmoves from the ice surface up towards the cameras. Hence, if multiple ofthese “minimum tracking volumes” were to simply be replicated andslightly overlapped, this would result in a considerable loss ofvaluable tracking region along the edges of the idealized square orrectangular volume. For all of these reasons, companies such as MotionAnalysis and Vicon have not approached the larger “tracking volume” as asuper-set of smaller volumes in what could be called a scalableapproach. For all practical purposes, each unique tracking volume shapeand size in combination with the expected number of objects (or players)requires a “custom” layout of tracking assemblies. As more and moreplayers continue to be added to the volume, more and more cameras willbe added based upon best judgments for placement. This entirearrangement could be best described as “player bound” as well as“quasi-volume bound.”

As previously discussed, the present inventors prefer the novel approachof separating the (X, Y) tracking of each player or game object as awhole from the (X, Y, Z) determination of each player's criticallocations (e.g., joints and body parts). Furthermore, the novel conceptof locating all of the identity markings on the “upper surface” of eachplayer facilitates the top-oriented field of view of the (X, Y) trackingassemblies which naturally limits inclusions due to player bunching.Given this separation of tasks, the present invention becomes highlyscalable. Each overhead (X, Y) area tracking assembly covers a fixed anduniform square or rectangular tracking area. Furthermore, the pixelresolution per this area remains substantially constant. To cover moretracking area, simply add additional (X, Y) assemblies. Each player thatmoves throughout this connected tracking area has their helmets (locatedat a minimum) and ideally also their shoulders. From this information,the player is identified along with pertinent information includingorientation, direction of movement, velocity, and acceleration as wellas current relative (X, Y) location. Using the current (X, Y) locationas well as the direction of movement, velocity and acceleration, thepreferred embodiment controllably directs one or more (X, Y, Z) pan,tilt and zoom cameras to automatically follow the given player.Furthermore, by intelligent inspection of the various projected playerpaths, the system optionally switches (X, Y, Z) cameras from one playerto another to best maximize overall tracking performance. By constantlyzooming each (X, Y, Z) camera to maximize player size per field of view,a uniform and ideal pixel resolution per marker square inch ismaintained. To cover more players, simply add additional (X, Y, Z) pan,tilt and zoom camera sets per added player.

D.2. The system should be minimally intrusive upon the objects to betracked and upon the surrounding environment especially if thatenvironment is a live setting.

As was previously discussed, beacon-based systems such as Trackus andOrad are environmentally transparent in terms of their microwavetracking energy but are intrusive in terms of their beacon technology.While these beacons are small and can conceivably be reduced further insize, they will still always be practically limited to locations wherethey can be embedded for both the safety and comfort of the player aswell as the protection of the device itself.

As was also discussed, vision-based systems such as Motion Analysis andVicon are intrusive in three key areas. First, their tracking energiesare in the visible spectrum such as red light that is visible to playersand fans alike. Second, their markers have substantial size and are bothnoticeable and prone to accidents in a full contact sport such as icehockey. And third, these same markers are covered with retroreflectivetape that is broad spectrum reflective including visible light. As such,existing rink illumination and portable lighting such as camera flashbulbs will tend to create unwanted and distracting reflections into theplayers' and fans' views.

And finally, the preferred embodiment of the present invention overcomeseach of these intrusions of the surrounding environment through itsnovel teachings as follows. First, the preferred tracking energy is inthe non-visible frequencies such as UVA or near IR. Second, the largespherical markers have been replaced with an “invisible” ink or paintthat can be adhered to the many necessary substrates while addingminimal thickness. And finally, this ink or paint, whether it bereflective, fluorescent or retroreflective in nature, is made“invisible” by its characteristic of only reflecting or emitting in thedesired non-visible frequencies.

D.3. The tracking signal-to-noise ratio should be maximized.

The microwave-based beacon systems such as Trakus and Orad will haveminimal noise due to ambient electromagnetic energies. However, due inpart to the enclosed nature of a hockey arena and more importantly tothe hard surfaces within the arena such as the ice surface itself, eachsignal transmitted from a player's helmet will bounce off these surfacescreating significant signal noise problems.

The vision-based systems such as Motion Analysis and Vicon attempt tolimit system noise by using a narrow band of tracking energy such as redlight with a corresponding camera filter. Unfortunately, by workingwithin the visible spectrum in their preferred approach, they aresusceptible to noise created by existing rink lighting, additionalportable lighting and the reflections these sources will cause offplayer jerseys and equipment. And finally, even if they work in thenon-visible IR region, both this IR energy and red light will reflectoff the ice surface creating false marker reflections that must be“mapped out” via software techniques creating “dead spots” within thetracking images.

The present invention is the only system to employ a combination oftracking energy absorbent and reflective compounds in order to “control”the tracking frequency noise. By using absorbers, unwanted reflectionsare significantly reduced or eliminated altogether. By using reflectivecompounds, unwanted reflection can be increased to a designatedintensity region that is detectably different from the marker signalintensity. These absorbent and reflective compounds can be applied toall of the background and foreground substrates. By first addressing thenoise issues, the foreground objects are preferably set at a minimal orzero reflection state while the background including the ice surface,the boards and the glass is preferably set at a narrow intensity rangedetectably different from the marker intensity.

D.4. Manufacturing and installation costs should be minimized and theresultant system should be simple for the user to maintain and operate.

To the extent that the present invention has been shown to be scalable,its manufacturing and installation parameters are more easily calculatedand maximized thereby leading to reduced costs. Furthermore, because ofits uniform scalable design, the present invention will minimize anyrequirements for operator intervention to resolve inclusions, lostplayer identity, or similar confusion due in general to signal-to-noiseproblems.

To the extent that the existing beacon- and machine vision-based systemsare not scalable, their manufacturing and installation parameters arenot as easily calculated and maximized thereby leading to increasedcosts. Furthermore, because these systems are not uniformly scalable incombination with the larger tracking volume and number of trackingmarkers associated with 3D multi-player movement, they are susceptibleto errors from inclusion and poor signal-to-noise ratios creating a needfor operator involvement.

Objects and Advantages

Accordingly, the objects and advantages of the present invention are to:

1—teach the fundamental component groups necessary for a multi-objectreal-time 3D object tracking system;

2—identify those individual components already in use within currentlyavailable systems and to which component groups they belong;

3—teach those novel components suggested by the present inventors inthis and their other four co-pending applications and to which componentgroups they belong;

4—teach how the present inventors' novel components allow currentlyavailable systems to better function in a live application with multiplecolliding objects, for instance a sporting event such as ice hockey;

5—identify the composition of currently available multi-object real-time3D object-tracking systems in terms of actual components used from eachgroup;

6—teach a novel preferred embodiment for a multi-object real-time 3Dobject-tracking system best suited for a live sporting event such as icehockey in terms of actual components used from each group;

7—teach several novel alternative embodiments using one or morecomponents of currently available systems mixed into the preferredembodiment; and

8—teach several novel variations using one or more of the presentinventors' novel components mixed into the currently available systems.

Further objects and advantages are to provide:

1—a system that is scalable and therefore comprises uniform assembliesthat are combinable into a matrix designed to increase tracking coveragein terms of area, volume and the number of objects while stillmaintaining uniform performance;

2—a system that is minimally intrusive upon the objects to be trackedand upon the surrounding environment especially if that environment is alive setting;

3—a maximized tracking signal to noise ratio; and

4—a system with maximized manufacturing and installation costs that issimple for the user to maintain and operate.

Still further objects and advantages of the present invention willbecome apparent from a consideration of the drawings and ensuingdescription.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram depicting all of the major componentsnecessary for the various multi-object tracking machine-vision systemsaccording to the present invention. These components are broken into tengroups including Camera Assembly, Tracking Frequency, Energy Source,Marker: Emission Method, Marker: Physical Form, Marker: ReflectiveShape, ID: Location, ID: Encoding Method, ID: Obtained, and CalibrationMethod.

FIG. 2 is a block diagram highlighting the combination of majorcomponents currently being used in several existing machine visionsystems with the main distinctive components of fixed (X, Y, Z) volumetracking cameras, visible light, spherical (attached ball) markers and afull body, unique constellation ID added by the present invention.

FIG. 3 is a block diagram highlighting the combination of majorcomponents in the preferred embodiment of the present invention,including the use of fixed (X, Y) area tracking cameras in combinationwith movable (X, Y, Z) volume tracking as well as non-visible IR or UL,flat reflective markers, and a top surface of body, encoded markings ID.

FIG. 4 is a block diagram highlighting the combination of majorcomponents in an alternate embodiment of the present invention. The maindistinction between this alternate and the preferred is the use of flatfluorescent markers instead of flat reflective.

FIG. 5 is a block diagram highlighting the combination of majorcomponents in another alternate embodiment of the present invention. Themain distinction between this alternate and the preferred is the use offlat retroreflective markers instead of flat reflective.

FIG. 6 is a block diagram highlighting the combination of majorcomponents in still another alternate embodiment of the presentinvention. The main distinction between this alternate and the preferredis that the ID is obtained during rather than outside game surfacetracking.

FIG. 7 is a block diagram highlighting the combination of majorcomponents in another alternate embodiment of the present invention. Themain distinction between this alternate and the preferred is the use offixed (X, Y, Z) volume tracking cameras rather than the fixed (X, Y)area tracking and movable (X, Y, Z) volume as well as flatretroreflective markers, and a full body, unique constellation ID.

FIG. 8 is a block diagram highlighting the combination of majorcomponents in an alternate embodiment of the present invention. The maindistinction between this alternate and the alternate of FIG. 7 is theuse of a top surface of body, encoded markings ID.

FIG. 9 is a block diagram highlighting the combination of majorcomponents in an alternate embodiment of the present invention. The maindistinction between this alternate and the alternate of FIG. 8 is thatthe ID is obtained during rather than outside game surface tracking.

FIG. 10 is a block diagram highlighting the combination of majorcomponents in another alternate embodiment of the present invention. Themain distinction between this alternate and the preferred is the use offixed (X, Y, Z) volume tracking cameras in place of the movable (X, Y,Z) volume.

FIG. 11 is a block diagram highlighting the combination of majorcomponents in another alternate embodiment of the present invention. Themain distinction between this alternate and the alternate of FIG. 10 isthe use of flat fluorescent rather than reflective markers.

FIG. 12 is a block diagram highlighting the combination of majorcomponents in another alternate embodiment of the present invention. Themain distinction between this alternate and the alternate of FIG. 11 isthe additional use of movable (X, Y, Z) volume tracking cameras as wellas flat retroreflective rather than reflective markers.

FIGS. 13 a and 13 b depict the theory and implementation of fixed (X, Y,Z) volume tracking camera assemblies.

FIGS. 14 a and 14 b depict the theory and implementation of fixed (X, Y)area tracking camera assemblies.

FIGS. 15 a and 15 b depict the theory and implementation of movable (X,Y, Z) volume tracking camera assemblies.

FIG. 16 a depicts the relationship between constant versus dynamicfield-of-view and its impact on the pixel resolution per player (object)tracked.

FIG. 16 b depicts how player bunching and therefore marker inclusion isaddressed by the use of movable (X, Y, Z) volume tracking cameraassemblies.

FIG. 17 is a side view drawing of a typical high intensity discharge(HID) lamp of the type often used to illuminated large open spaces suchas a sporting arena or facility, further depicting the spread of emittedelectromagnetic frequencies ranging from UV through visible light intoIR.

FIG. 18 a is a side view of the same HID lamp showing its emitted energybeing dispersed in multiple directions as it strikes a typicalreflective material.

FIG. 18 b is a side view of the same HID lamp showing its emitted energybeing redirected back towards the lamp as it strikes a typicalretroreflective material.

FIG. 18 c is a side view of the same HID lamp showing its emitted energybeing fluoresced and then dispersed in multiple directions back towardsthe lamp as it strikes a typical fluorescent material.

FIG. 19 is a side view of a typical HID lamp, further depicting thespread of emitted electromagnetic frequencies ranging from UV, throughvisible light into IR. Also shown are three variably orientedretroreflective elements partially embedded in a single binder that hasbeen joined to a substrate. The elements and binder have been depictedas transmissive to visible light while the substrate is reflective. Inresponse to the non-visible frequencies of either UV or IR, at leastsome of the elements are retroreflective while the substrate remainsreflective.

FIG. 20 a depicts a spherical retroreflective marker and its circularreflection while FIG. 20 b depicts a hockey player with attachedspherical markers.

FIG. 21 a is a set of three perspective drawings depicting a typicalplayer's jersey, typical player's pads with tracking patches in place,and then a combination of the jersey over the pads with patches.

FIG. 21 b is a set of two perspective drawings depicting a hockey puckas well as a typical player's hockey stick, where each has beenaugmented to include tracking ink on at least some portion of its outersurfaces.

FIG. 21 c is a set of two perspective drawings depicting a typicalhockey player's helmet which has been augmented to include trackingstickers on at least some top portion of its outer surface.

FIG. 22 a depicts a hockey player set up with spherical markers whileFIG. 22 b shows the resultant circular reflections that will be seenwith an appropriate vision system.

FIG. 22 c depicts a hockey player set up with flat markers while FIG. 22d shows the resultant multi-shape reflections that will be seen with anappropriate vision system.

FIG. 23 a depicts two different hockey players set up with sphericalmarkers. In the depicted view, the players are not overlapping. FIG. 23b shows the resultant circular reflections that will be detected byappropriate frame analysis.

FIG. 23 c depicts the same two different hockey players that were shownin FIG. 23 a except that they are now overlapping. FIG. 23 d shows theresultant circular reflections that will be detected in this case.

FIG. 24 depicts the combination of encoded ID marks as well as trackmarks that are preferably placed upon a helmet sticker. Also shown bycomparison is the difference between the “top surface” helmet IDapproach versus the full body “constellation” ID approach.

FIG. 25 depicts the separation of player identification and tracking onthe non-playing surfaces of the entrance and exit passageway and theteam benches versus player tracking only (and not identification) on theplaying surface.

FIG. 26 a depicts the pre-tracking calibration method most typicallyused with current fixed (X, Y, Z) volume tracking camera-based systems.

FIG. 26 b depicts the dynamic calibration method taught for use with thefixed (X, Y) area and movable (X, Y, Z) volume tracking cameras of thepreferred embodiment.

FIG. 27 a depicts a typical hockey player's pads, helmet, stick and puckbeing captured from an overhead X-Y filming camera and displayed on aviewing screen.

FIG. 27 b is similar to FIG. 27 a except that now all of the foregroundobjects have been first treated with an energy-absorptive compound afterwhich tracking marks have been added to desired locations.

FIGS. 28 a and 28 b depicts the alternate embodiment that uses fixed (X,Y) area cameras for player identification and tracking on thenon-playing surfaces while also using only movable (X, Y, Z) volumetracking cameras to track the players on the playing surface. Thereforethe fixed (X, Y) area cameras are not first being used to track theplayers on the playing surface in order to direct the movable camerasbut rather a predictive technique is employed to guide the movablecameras as they follow their intended targets.

FIG. 29 a depicts a potential shoulder mark that along with the jerseyit is placed upon can potentially create up to three differentreflectivity intensity levels depending upon the use of absorbent,reflective and retroreflective compounds.

FIG. 29 b depicts the strategic regular placement of markings upon theshaft of a hockey stick that allow the stick to serve as a dynamiccalibration tool similar to the preferred track mark on the player'shelmet.

FIG. 30 depicts the additional strategic regular placement of markingsupon the playing surface such as the boards and channels that hold theglass enclosing the hockey rink. These markings further serve as adynamic calibration tool for use especially by the movable (X, Y, Z)volume tracking cameras.

DETAILED DESCRIPTION OF THE INVENTION

Referring first to FIG. 1, there is shown a block diagram depicting allof the major components 1000 anticipated to be necessary for the variousmulti-object tracking machine vision systems according to the presentinvention. These components are broken into ten groups including CameraAssembly 500, Tracking Frequency 510, Energy Source 520, Marker:Emission Method 530, Marker: Physical Form 540, Marker: Reflective Shape550, ID: Location 560, ID: Encoding Method 570, ID: Obtained 580 andCalibration Method 590. Camera Assembly 500 can be one or more of Fixed(X, Y, Z) Volume Tracking assemblies 502, Fixed (X, Y) Area Trackingassemblies 504 and Movable (X, Y, Z) Volume Tracking assemblies 506.Tracking Frequency 510 can be one or more of Visible Light 512, InfraredLight 514 or Ultraviolet Light 516. Energy Source 520 can be one or moreof Ring Lights Emitting Visible or IR Frequencies 522, Existing LightsEmitting Visible Frequencies 524 and Existing Lights Modified to EmitNon-Visible Frequencies 526.

Within all major components 1000, there are three characteristics ofmarkers that are categorized as follows. The possible Marker: EmissionMethod 530 is retroreflective 532, reflective 534 or fluorescent 536.The possible Marker: Physical Form 540 is spherical (attached ball) 542or flat (embedded/applied ink) 544. And finally, the Marker: ReflectiveShape 550 corresponding to the Marker: Physical Form 540 is uniformcircular 552 or non-uniform multi-shape 554, respectively.

Within components 1000, there are three characteristics of ID(identification markers) categorized as follows. The possible ID:Location 560 is on the full body 562 or top surface of body 564. Thepossible ID: Encoding Method 570 corresponding to the ID: Location 560is a unique constellation 572 or encoded markings 574. And finally, thepossible ID: Obtained 580 is during game surface tracking 582 or outsideof game surface tracking 584. The last group of components 1000 is theCalibration Method 590 that can be either pre-tracking 592 orsimultaneously with tracking 594.

Referring now to FIG. 13 a, there is shown an example of a fixed (X, Y,Z) volume tracking camera 502 that comprises a camera 126, filter andconnection to a local computer system for video processing and analysis160. Camera 126 can be one of any analog or digital-imaging cameras astypically used for industrial vision applications. One example is theEagle digital camera used by Motion Analysis Corporation that features aceramic metal oxide semiconductor (CMOS) image sensor with 1280×1024pixel resolution and a maximum capture rate of 600 million pixels persecond. It is important to note that, once in place, this volumetracking camera 126 has a fixed field-of-view (FOV) similar to afour-sided pyramid in shape within an image cone 121 v. The actual pixelresolution per inch of the FOV will vary throughout the height 121 h ofthe pyramid ranging from a higher value at the top width 121 tw to alower value at the bottom width 121 bw. These cameras are typicallysecured from an overhead position to have a perspective view arrangement502 m of the desired tracking volume as shown in FIG. 13 b.

Referring now to FIG. 13 b, there is shown one particular arrangement offixed (X, Y, Z) cameras 502 that, when taken together, form a uniquelyshaped tracking volume through which a player 17, wearing markers suchas spherical markers 17 sm, may transverse. The resultant resolution percross-sectional area of this volume 121 tv is non-uniform. For example,while skating through any given point in the tracking volume, markers 17sm on one body part of player 17 may be viewed by camera 126 e with amuch lower resolution per inch than similar markers on a different bodypart. Also, the second camera such as 126 d may have a much differentpixel resolution of marker 17 sm than camera 126 e. Cameras 126 a, 126 band 126 c may each have obstructed views of marker 17 sm.

Referring now to FIG. 14 a, there is shown an example of fixed (X, Y)area tracking camera 504, that comprises a tracking camera 124 with afilter 124 f that have been enclosed in a protective housing 121 with atransparent underside 121 a. Also enclosed in housing 121 is an energysource 10 emitting tracking energy 11 as well as unfiltered filmingcamera 125. Tracking camera 124 and filming camera 125 are connected toa local computer system for video processing and analysis 160. Theentire assembly included within housing 121 is preferably secured in anoverhead position looking directly down at a subset of the trackingsurface. From this overhead position, camera 124 has a fixed FOV 120 vthat is focused on the top surface of any players below and as suchmaintains a substantially uniform pixel resolution per tracking area FOV120 v.

Referring now to FIG. 14 b, there is shown a scalable area trackingmatrix 504 m comprising multiple fixed (X, Y) area tracking cameras 120c aligned such that their FOVs 120 v are substantially side-by-side witha small overlap for calibration purposes. Throughout this scalablematrix 504 m, the top surface 110 of player 17 can be readily tracked.

Referring now to FIG. 15 a, there is shown an example of movable (X, Y,Z) volume tracking camera 506, that comprises a pan, tilt and zoomcamera 140 with a filter that is connected to local computer system forvideo processing and analysis 160. Top surface 110 of player 17 is heldin constant view by one or more of cameras 140 that are controllablypanned, tilted and zoomed for maximum desirable pixel resolution perplayer. The information for this controlled movement is based eitherupon the current (X, Y) coordinates of player 17 as previouslydetermined from information gather by scalable area tracking matrix 504m or by movement tracking algorithms calculated by computer 160 topredict the next possible location of player 17.

Referring now to FIG. 15 b, there is shown a scalable volume trackingmatrix 506 m comprising multiple movable volume tracking cameras 506where one or more cameras form an assembly and are dynamically assignedto a player 17. As will be explained in more detail using FIGS. 16 a and16 b, this dynamic process of automatically panning, tilting and zoomingeach movable camera to maintain the maximum desirable pixel resolutionper player provides a substantial benefit over the arrangement of fixedvolume tracking cameras 502.

Referring now to FIG. 16 a, there is shown a series of three viewsdepicting the top surface 110 x of player 17 (i.e., player 17 withabsorbers and markings applied) at close range, mid-range and far rangewith respect to a fixed volume tracking camera 126. As can be seen, thepixel resolution per the player's body surface area is substantiallydifferent between close and far ranges. However, FIG. 16 a also shows aseries of three views the same player 17 and relative locations but nowwith respect to movable volume tracking camera 140. As can be seen, thepixel resolution per the player's body surface area is now substantiallyuniform.

Referring now to FIG. 16 b, there is shown an example matrix of fourFOV's 120′ created by area tracking cameras 124. Within this combinedgrid, several players having top surfaces such as 110 x and 111 x movefreely about. In this particular example, four movable cameras 140-a,140-b, 140-c and 140-d are tracking the player with top surface 110 x.As depicted, the FOV's for cameras 140-b and 140-d are almost fullyblocked by other players whereas the FOV for camera 140-a is partiallyblocked but the FOV for camera 104-c is clear. The preferred embodimentwill automatically reassign cameras such as 141-d that may already betracking another player, (e.g., the player with top surface 111 x) tonow follow a different player with top surface 110 x so as to ensuretotal maximum player visibility. This reassignment decision can be basedupon the information gathered by the scalable area tracking matrix 504m, predictive calculations made by computer 160 concerning the expectednext positions of any and all players, or both.

Referring now to FIG. 17, there is shown an example of three differentTracking Frequencies 510 being emitted by normal or modified HID lamp10. These include UV ray 11, visible ray 12 and IR ray 13. As these rays11, 12 and 13 strike reflective material 20 attached to substrate 30,they will cause reflected UV ray 11 r, visible ray 12 r and IR ray 13 r.

Referring now to FIGS. 18 a, 18 b and 18 c, there is shown an example ofthree different Marker: Emission Methods 530 caused by reflectivematerial 20 a, retroreflective material 20 b and fluorescent material 20c. In FIG. 18 a, lamp 10 emits rays 11, 12 and 13 which are thenreflected off reflective material 20 a in a diffuse manner causing raysr1. In FIG. 18 b, emitted rays 11, 12 and 13 are retroreflected offretroreflective material 20 b in a manner causing rays r2. In FIG. 18 c,emitted rays 11, 12 and 13 are first absorbed by fluorescent material 20c causing emitted rays r3. Reflective material 20 a and fluorescentmaterial 20 c have an advantage over retroreflective material 20 b inthat their reflected and fluoresced rays r1 and r3, respectively, willhave a wider viewing angle than retroreflected rays r2. Retroreflectivematerial 20 b has an advantage over materials 20 a and 20 c because itsrays r2 will be of stronger combined energy for a longer distance.Fluorescent material 20 c has an advantage over materials 20 a and 20 bbecause it can absorb visible light readily available in largerintensities within the ambient environment and convert this to anon-visible tacking energy such as IR.

Referring now to FIG. 19, there is shown a novel retroreflectivematerial 100 that is similar to commercially available cube-cornered ormicrobead retroreflectors except that it has been modified to betransparent to any energies that are not intended to be retroreflected.In the case where the tracking frequency 510 is chosen to be UV light516, HID lamp 10 is shown to emit UV ray 11 that enters retroreflectiveelement 20 uv that is coated with UV reflector 24 uv. Reflector 24 uvthen reflects ray 11 back up through element 20 uv becomingretroreflected ray 11 r. Visible ray 12 and IR ray 13 will pass throughreflector 24 uv. In the case where the tracking frequency 510 is chosento be IR light 514, HID lamp 10 is shown to emit IR ray 13 that entersretroreflective element 20 ir that is coated with IR reflector 24 ir. IRReflector 24 ir then reflects ray 13 back up through element 20 irbecoming retroreflected ray 13 r. Visible ray 12 and UV ray 11 will passthrough IR reflector 24 ir. Retroreflective elements 20 uv and 20 ir areembedded within binder 28 that is attached to substrate 30. Binder 28 issubstantially transparent to UV ray 11 a and IR ray 13 a.

Referring now to FIG. 20 a, there is shown an example of the first ofthe two Marker: Physical Forms 540, namely spherical (attached ball)542, also referred to as 17 sm. Spherical marker 17 sm comprises aretroreflective sphere 17 s that is attached to a base 17 b. Companiessuch as Motion Analysis and Vicon currently use this type of marker. Thetypical retroreflective sphere 17 s retroreflects a broad spectrum offrequencies including UV ray 11, visible ray 12 and IR ray 13 causingretroreflective rays 11 a, 12 a and 13 a respectively. Theseretroreflective rays 11 a, 12 a and 13 a then create resulting circularimage 17 c that is incident upon any tracking cameras such as 124, 126and 140. Image 17 c is an example of one of the two Marker: ReflectiveShapes 550 for a marker, namely uniform circular 552.

Referring now to FIG. 20 b there is shown an example of the first of twoID:Locations 562 for the player ID, namely full body 562. In FIG. 20 b,spherical markers 17 sm are placed at various key locations over theentire body of player 17. For practical purposes retroreflective tape 17t is used to cover the blade of stick 104.

Referring now to FIGS. 21 a, 21 b and 21 c, shown are several examplesof the second of two Marker: Physical Forms 540, namely flat(embedded/applied ink) 544. In FIG. 21 a, right and left trackingpatches 107 r and 107 l are shown attached to player shoulder pads 106that are typically covered by jersey 105. Patches 107 r and 107 l havebeen pre-marked with special ink formulated to reflect, retroreflect, orfluoresce only the desired tracking energy. Such pre-markings includeorientation marks 107 r 1 and 107 l 1 as well as bar code marks 107 r 2and 107 l 2. In FIG. 21 b, puck 103 has been coated with similar specialink 103 a while the blade of stick 104 has been wrapped with reflectivetape 104 a. And finally, in FIG. 21 c, sticker 109 has been applied tohelmet 108 and comprises a uniquely identifying mark created withsimilar special ink. Each of FIGS. 21 a, 21 b and 21 c illustrateretroreflected ray 11 r.

Referring now to FIG. 22 a, the information depicted in FIG. 20 b isrepeated to dramatize FIG. 22 b that depicts the image formed incomputer 160 based upon uniform circular 552 reflections. This distinctformation of marker reflections 17 c can be used to identify player 17and is the first of two ID:Encoding Methods 570 called uniqueconstellation 572. Companies such as Motion Analysis and Vicon use theunique constellation 572 method for identifying human objects such asplayer 17. Furthermore, these same systems are designed to identify thehuman object while they are also tracking their motion. This is thefirst way the player ID is Obtained 580, namely during game surfacetracking 582.

Referring now to FIG. 22 c, there is shown information similar to FIGS.21 a, 21 b and 21 c to dramatize FIG. 22 d that depicts the image formedin computer 160 based upon non-uniform multi-shape 554 reflections. Thiscollection of individual markings 17 m that have been placed at variouslocations on player 17 are only used to locate a particular body partand its orientation rather than to identify the player 17. In thepreferred embodiment that employs these types of flat 544 markings, theidentification of player 17 is based upon a top surface of the body 564Id Location 560.

Referring now to FIGS. 23 a, 23 b, 23 c and 23 d there is dramatized theproblems inherent with full body 562 unique constellation 572 playeridentification. In FIG. 23 a, there is shown two players 17 and 18 thatare each pre-marked with a unique constellation of spherical markers 17sm. The view of players 17 and 18 is not overlapping in FIG. 23 a. Theresultant image detected by computer 160, namely of circular reflections17 c and 18 c, is shown in FIG. 23 b. Reflections 17 c and 18 c are alsonot overlapping. Referring now to FIG. 23 c, players 17 and 18 are nowoverlapping causing the resultant overlapping of reflections 17 c and 18c as shown in FIG. 23 d. Note the considerably more difficultidentification problem presented to computer 160 as players such as 17and 18 begin to block each other's view in one or more volume trackingcameras such as 126 or 140.

Referring now to FIG. 24, there is shown a set of preferred helmetstickers 109 id 64, 109 id 00, 109 id 14 and 109 id 13 implementing theuniquely encoded markings 574 method of ID:Encoding Method 570. Themarkings on stickers 109 id 64, 109 id 00, 109 id 14 and 109 id 13 arecreated using the special ink formulated to reflect, retroreflect orfluoresce preferably only the chosen Tracking Frequency 510. The playerid is preferably implemented as a traditional bar code and could beembedded on the helmet stickers 109 id 64, 109 id 00, 109 id 14 and 109id 13 in a non-visible IR or UV reflective, retroreflective orfluorescent ink. Also depicted is helmet sticker 109 tm that includes aspecial tracking mark designed to help computer 160 both locate helmet108 as well as determine its orientation. This special tracking mark maybe created using either a non-visible IR or UV reflective,retroreflective or fluorescent ink. Sticker 109 id&tm combines both theid marks as well as the tracking marks, and can either be created usingthe same non-visible frequency, such as both IR or both UV, or differentfrequencies, such as one IR and the other UV. Note the considerablysimpler identification problem presented to computer 160 as it analyzeshelmet stickers such as 109 id&tm viewed by area tracking cameras 124.Cameras such as 124 are looking down upon the top surface of the bodiesof players such as 17 and 18 and are therefore not expected toexperience information degradation due to player overlapping. Stickers109 id 64, 109 id 00, 109 id 14 and 109 id 13, 109 tm and 109 id&tm arerepresented generically as sticker 109 that is shown attached to helmet108.

Referring now to FIG. 25, there is shown the second way in which theplayer ID is Obtained 580, namely outside of game surface tracking 584.Rink entrance and exit 12 e as team bench 12 f are in constant view ofone or more area ID & tracking cameras similar to 124 except with anarrowed FOV 122 v. Narrowing FOV 122 v provides an increased pixelresolution per inch when looking down upon the players' helmets 108 andattached stickers such as 109 tm&id. The increased pixel resolutionallows for more complex encoding, i.e. patterns with smaller markings onthe limited space of the helmet sticker 109. The rink playing surface102 is in constant view of the scalable area tracking matrix 504 mcomprising multiple cameras 124 with normal FOV's 120 v.

Also shown are a single set of four movable volume tracking cameras140-a, 140-b, 140-c and 140-d that are for example currently assigned totrack the top surface 110 of a player starting when he first enters theplaying surface 102 from the entranceway 12 e. Tracking with cameras140-a, 140-b, 140-c and 140-d continues as the player transversessurface 102 and ceases when player exits surface 102 and enters teambenches 12 f. Once within bench area 12 f, area ID & tracking camerassimilar to 124 track the player and also reconfirm the player's identityby viewing helmet sticker 109 tm&id. At any time, the player maysubsequently leave bench area 12 f and reenter surface 102 where againhis motion is tracked by movable volume cameras 140-a, 140-b, 140-c and140-d. Eventually, the player will either exit the playing area throughentrance and exit 12 e or return again to bench 12 f and be tracked andre-identified by the ID & tracking cameras.

Referring now to FIG. 26 a, there is shown the first type of CalibrationMethod 590, namely pre-tracking 592. Companies such as Motion Analysisand Vicon currently perform this method in order to calibrate theirfixed volume tracking cameras 126 after they have been set into place.The calibration tool 130 comprises two or more markers such asvarious-sized spherical balls 17 sm whose dimensions are pre-known andthat are affixed on the tool 130 at pre-known distances from each other.The calibration process begins when tool 130 is held up within the FOVof two cameras 126. Computer 160 receives images from each of thesefirst two cameras and processes individually the reflected circles fromthe calibration tool 130. Using stereoscopic algorithms that are wellknown in the art, the locations of each spherical marker 17 sm on tool130 are calculated within a local coordinate system. The operatorholding the tool 130 then moves it into the view of a third camera 126while still being in view of one of the two prior cameras 126. Thistechnique is continued until all of the fixed cameras 126 have beenindividually added to the calibration of all previous cameras 126.

The present inventors anticipate that this same technique, although itwould not be ideal, could be used to pre-calibrate the scalable areatracking matrix 504 m. In consideration of area matrix 504 m, therelative orientation of each camera 124 is primarily side-by-side withits neighbors, allowing for a small overlap on the edges of its FOV 120v. Furthermore, the preferred orientation of FOV 120 v is “top down,”rather than the “perspective” view of cameras 126. Given thesearrangements, a preferable pre-tracking calibration technique would beto use a traditional calibration plate incorporating a fixed set ofmarkings held at pre-known distances from each other. This plate wouldthen be held in a fixed position facing up at the junctions betweenevery two cameras 124 overlapping FOV's 120 v. Again, using standardtechniques well known in the art, each of the two cameras could then bejointly calibrated by computer 160. Proceeding throughout all camerajunctions in the same fashion would complete the calibration of thenetwork to itself. The only remaining task would be to calibrate theentire matrix 504 m to the playing surface 102, entrance and exit 12 eand team benches 12 f. This could be accomplished by placing a markingat a fixed pre-known location somewhere within each of the areas ofsurface 102, entrance and exit 12 e and team benches 12 f. Once capturedby computer 160 through one or more cameras within matrix 504 m, thesemarkings at pre-known locations would serve to register the entirematrix.

Now referring to FIG. 26 b, there is shown the second type ofCalibration Method 590, namely simultaneously with tracking 594. Thisprocess begins after the scalable area tracking matrix 504 m is itselfpre-calibrated as described in the previous paragraph. Once eachoverhead camera 124, within assembly 120 c has been calibrated, it willbe used as the basis for the dynamic re-calibration of movable cameras140 as they continually change their orientation and FOV. Aftercalibration, each camera 124 will have a fixed (X, Y) coordinate systemregistered with the playing surface 102, entrance and exit 12 e and teambench 12 f. Calibration simultaneous with tracking 594 begins when aplayer 17 enters the view of at least one area tracking camera 124 andis therefore detected by computer 160. The markings that computer 160will be viewing based upon camera 124 will be those on the top surfaceof the body 574 including the helmet sticker 109 tm&id. Stickers such as109 tm&id are similar to calibration tool 130 in that their markings arepre-known in both size and orientation to each other.

As depicted in FIG. 26 b, at least one point on sticker 109 tm&id thatis in view of both fixed pre-calibrated camera 124 and movable camera140 is first located in local rink (X, Y) coordinates based uponinformation provided by camera 124. Once located, the same point isanalyzed by computer 160 from the images captured by camera 140 alongwith other measurable information such as the current rotations of thepanning and tilting mechanisms supporting camera 140 as well as thezooming mechanism associated with its lens. During analysis, thedetermined (X, Y) location of the captured point is used to center the(X, Y, Z) coordinate system of camera 140. Once centered, the (Z) heightscale can be set and then used to apply to all other common points inview of both the (X, Y) camera 124 and the (X, Y, Z) camera 140. Thesepoints include not only those on helmet sticker 109 tm&id but also thosethroughout all the body of player 17.

Furthermore, it is expected that additional volume cameras 140 assignedto track the same player 17 will similarly be simultaneously calibratedwith camera 124. It should be noted that player 17 may be straddling aboundary between area tracking cameras 124 and as such two differentvolume cameras 140 may actually be calibrated for the same player 17 bytwo different area cameras 124. In practice, this is immaterial sincethe pre-calibration by system 160 of the entire scalable area trackingmatrix 504 m can be thought of as creating one large single area (X, Y)tracking camera. Hence, it can be seen that each of the volume camerassuch as 140 in the present figure or 140-a, 140-b, 140-c and 140-d ofprior figures that are currently assigned to follow player 17 aresimultaneously calibrated frame-by-frame to the overhead matrix 540 m.Furthermore, once calibrated the multiple cameras such as 140-a, 140-b,140-c and 140-d may be used to stereoscopically locate markings onplayer 17 that are not in view of the overhead matrix 540 m.

Referring now to FIG. 27 a, there is shown an alternate embodiment 120 bto area (X, Y) tracking camera assembly 120 c that does not includeadditional overlapping filming camera 125. In this alternate embodiment,enclosure 121 houses lamp 10 and tracking camera 124 with visible lightfilter 124 f and is enclosed on the bottom surface by transparent cover121 a through which tracking energy 11 may transmit. Alternateembodiment of area tracking camera assembly 120 b is also connected tocomputer 160 (not depicted) that is in turn connect to video terminal127 via cable 121 c. Further shown is player 17 to which trackingpatches 107 r and 107 l and helmet sticker 109 have been attached. Alsoshown are puck 103 with reflective ink 103 a and stick 104 with ink 104a. Shown on terminal 127 is camera image 128 that includes player 17.The body of player 17 is portrayed as dimmed due to some reflectance ofthe non-visible tracking energy while the patches, stickers and ink areportrayed as white due to their higher engineered reflectance.

Referring now to FIG. 27 b, there is shown an identical arrangement toFIG. 27 a except that player 17 has been first treated with one or moretracking energy absorbent compounds after which tracking patches,stickers and inks were applied. Similarly, stick 104 has also been firsttreated. As such, player 17 has now become 17 while stick 104 has become104 t. Due to this novel application of energy absorbers, treated player17 is no longer visible on terminal 127. Terminal 127 displays cameraimage 128 provided by computer 160 in response to the images captured bythe various tracking cameras 124, 126 and 140.

The present inventors have listed many absorbers and treatments that maybe used especially to absorb UV frequencies in the prior co-pendingapplication entitled Employing Electromagnetic By-Product Radiation inObject Tracking. Such treatments and absorbers are also well known forthe IR frequencies to someone skilled in the art. Further, the presentinventors have shown that it may also be similarly beneficial to applyenergy reflectors rather than absorbers, especially with respect to thebackground such as player surface 102. What is important is that theintensity of the reflected signal off the tracking marks be clearlydistinguishable from any reflections off the background or foreground(player's body and equipment). To gain this clarity of signaldifferentiation, it may be desirable to either reduce reflectionsthrough absorption or to increase reflection through reflectivematerials. This concept of absorbers and reflectors for the control ofthe signal-to-noise ratio was not listed in system 1000 as a separatecomponent since the present inventors see it as a beneficialoptimization to every possible combination of components listed in 1000.

Referring now to FIGS. 28 a and 28 b, there is shown an advantageousnovel modification to the preferred embodiment that employs a scalablearea-tracking matrix 504 m along with a scalable movable volume-trackingmatrix 506 m. Specifically, that portion of the scalable area-trackingmatrix 504 m that was in place to track players such as 17 while theymoved about the playing surface 102 has been eliminated. The reductionrepresents a saving in system, installation and maintenance costs.

Referring specifically to FIG. 28 a, this is made possible by theunderstanding that as player 17 with top surface 110 passes throughentrance 12 e he will first still be viewed by the tracking cameras 124left in place to cover this area, through their FOV's 122 v. Thesecameras will first identify player 17 and then follow his movements upuntil he enters player surface 102. As player 17 enters surface 102, thecomputer 160 will automatically direct cameras 140-a, 140-b, 140-c and140-d to pick up player 17.

Referring specifically to FIG. 28 b, as player 17 with top surface 110is first viewed, computer 160 will be constantly calculating andrevising its prediction of the player's next movements and thereforewhereabouts. Cameras 140-a, 140-b, 140-c and 140-d will continuouslypan, tilt and zoom to follow the travel of player 17. Eventually, player17 will leave surface 102 and enter team bench 12 f where he will againbe in the view of tracking cameras 124 left in place to cover this area,through their FOV's 122 v. Computer 160 will constantly monitor player17 even while he remains on team bench 12 f. At some point throughoutthe competition, player 17 is expected to re-enter the playing surface102. At this time computer 160 will automatically direct cameras 140-a,140-b, 140-c and 140-d to follow the player's travel until he eitherreturns to team bench 12 f or leaves through entrance and exit 12 e.

Referring now to FIG. 29 a, there is shown another advantageousmodification to the preferred embodiment for marking foreground objectssuch as jersey 105. Specifically, there are three distinct areas of theforeground object for which it is desirable to have clearlydistinguishable reflected intensity levels of the tracking energy UV orIR. The first area is the jersey 105 itself. Second, there is the base107 of tracking patch 107 r and third, there is the tracking mark 107 cof patch 107 r. Correspondingly, there are three appliques described inthe present invention that can create the desired distinguishablereflected intensity levels: (1) UV and IR absorbent compounds, (2) UVand IR reflective compounds, and (3) UV and IR retroreflectivecompounds. One possible arrangement of these compounds is to first applythe absorber to jersey 105, the reflective compound to base 107 and theretroreflective compound to tracking mark 107 c. Two other combinationsthought to be particularly useful are: (1) Using the retroreflectivecompound on base 107 and using more absorber for mark 107 c; and (2)Using the reflective compound on jersey 105, the absorber on base 107and the retroreflective compound on mark 107 c. Other combinations areanticipated in combination with the teachings of the present invention.

Referring now to FIG. 29 b, there is shown another advantageousmodification to the preferred embodiment for creating a dynamiccalibration tool that can be used to help calibrate the movable volumetracking matrix 506 m simultaneously with tracking 594. Specifically,precisely measured and spaced track markings 104 m have been placed ontostick 104. The exact size, shape and spacing of these markings 104 m areimmaterial to the concept being presently taught. Placing these markingsupon a rigid object that is used by each player during the game willprovide computer 160 with a way to verify its calibration estimates ofboth fixed area tracking camera 124 and especially movable volumecameras 140. Another possibility is to place similar markings onto thepipes of the goals on either end of the playing surface.

Referring now to FIG. 30, there is shown a further advantageousmodification to the preferred embodiment for assisting in the dynamiccalibration of movable volume tracking matrix 506 m. Rink playingsurface 102 is shown in perspective view in between near boards 103 nband far boards 103 fb. Attached to near boards 103 nb are glass supportcolumns such as 105 nc that are holding in place glass panes such as 105ng. Attached to far boards 1031 b are glass support columns such as 105fc that are holding in place glass panes such as 105 fg. Placed on boththe outer (shown) and inner (not shown) surface of near support columns105 nc are reflective markings 105 nm. Similarly, placed on both theinner (shown) and outer (not shown) surface of far support columns 105fc are reflective markings 105 fm. Also shown are markings 103 fm on theinner side of far boards 103 fb. Similar markings on the inner side ofnear boards 103 nb are also anticipated. Moving about on surface 102 areplayers 16 and 17 that are being constantly tracked by movable cameras140-b and 140-d as well as 141-b and 141-d, respectively. During playertracking as cameras such as 140-b, 140-d, 141-b and 141-d pan, tilt andzoom to change their FOV's, they will be constantly picking up one ormore reflective marks such as 105 nm, 105 fm and 103 fm. The exact size,shape and spacing of these markings 105 nm, 105 fm and 103 fm areimmaterial to the novel concept being presently taught. Placing thesemarkings upon at least the locations specified will provide computer 160with a means of verifying its calibration estimates of movable volumecameras 140.

Summary of Optimized Systems

A careful study of the for prior co-pending applications filed byapplicants and identified above along with the previously describedcomponents 1000 will suffice to teach those skilled in the art how eachcomponent may operate as a functional part of a complete system.Therefore, the remainder of this application will focus ondistinguishing at a higher level the various possible novel optimizedsystems according to the present invention along with discussions as totheir tradeoffs.

Referring now to FIG. 2, there is shown a block diagram depicting all ofthe major components 1000 of which a subset has been identified asrepresentative of real-time 3D-object tracking system 1002. System 1002comprises fixed (X, Y, Z) volume tracking assemblies 502 that employeither visible light 512 or IR light 514 that is emitted from ringlights 522. This tracking energy is then reflected by spherical(attached ball) 542 retroreflective 532 markers that are placed atvarious locations on the subject to be tracked and that create a uniformcircular 552 image in camera assemblies 502. The full body 562 set ofspherical markers 542 form a unique constellation 572 used by system1002 to identify each subject during game surface tracking 582. Thesystem 1002 is calibrated prior to tracking 592.

While system 1002 accomplishes the goals of real-time 3D tracking, ithas several drawbacks as follows:

1—Due to its non-uniform approach to camera placement, it is difficultto scale up to track larger areas such as a hockey rink;

2—Due to its strategy of fixed volume tracking accomplished with acomplex overlapping network of camera field-of-views, the pixelsresolution per player is inconsistent and the system is prone to losemarkers when multiple players bunch up;

3—When using visible light as the tracking energy, the additional redlight is added by the system because of the need to use ring lights incombination with the retroreflective markers creating a lighting systemthat is intrusive to both players and audience;

4—When using in IR light as the tracking energy, the ring lights do notemit any additional visible light in combination with the IR to act as acue to both players and audience not to stare at the lighting;

5—The system employs retroreflective markers in order to obtain thehighest possible signal reflection but these materials are broad-bandreflectors that respond to the entire visible spectrum causing unwantedreflections of ambient light sources that is intrusive to both playersand audience;

6—The retroreflective markers are constructed to be spherical balls thatprotrude away from the body helping to ensure maximum visibility to thetracking cameras by consistently creating a circular reflection from anyangle, however, this very nature of their protruding physical form makesthem vulnerable to dislodge during player contact;

7—The uniform circular nature of the retroreflection caused by thespherical marker is useful for centroid calculations and thereforedetermining exact body points, however, it necessarily forces morecameras since less player surface area can be marked for viewing using aspherical shape;

8—By attempting to combine player identification with body jointtracking the system creates a difficult requirement that forces cameraviews away from the top down view that rarely experiences inclusions dueto player bunching to a perspective view that is very susceptible toinclusions;

9—By attempting to combine player identification with body jointtracking the system creates a difficult requirement that substantiallyall markers placed on the full body must be in view in order to identifya given player;

10—By attempting to combine player identification with body jointtracking the system loses an opportunity to perform playeridentification off the game surface in either the player entrance andexit or on the team benches. (These areas are short on space foradequate perspective camera placement and are also very crowded withplayers who in the case of team benches are expected to be sittingtherefore additionally hiding markers);

11—By attempting to combine player identification with body jointtracking the system (a) creates a difficult requirement that each playerhave either a substantially different body shape and thereforeconfiguration of markers or that additional markers be added to create aunique pattern, and (b) makes using the system with up to twenty someplayers per team cumbersome as a pre-tracking procedure must beinstituted to ensure adequate “constellation” differentiation perplayer; and

12—The system has no method of simultaneously calibrating the camerasduring tracking that precludes the possibility of using movable volumetracking cameras that could dynamically reconfigure the tracking volumeto better create a uniform pixel resolution per player and reduce thenumber of marker inclusions due to player bunching.

Referring now to FIG. 3, there is shown a block diagram depicting all ofthe major components 1000 of which a subset has been identified as thepreferred embodiment 1004 of the present invention. Preferred embodiment1004 comprises fixed (X, Y) area tracking assemblies 504 in combinationwith movable (X, Y, Z) volume tracking assemblies 506, that employeither IR light 514 or UV light 516 that is provided by existing lightsmodified to emit non-visible frequencies 526. This tracking energy isthen reflected by flat (embedded/applied ink) 544 reflective 534 markersthat are placed at various locations on the subject to be tracked andthat create non-uniform multi-shape 552 images in camera assemblies 504and 506. Specially encoded 574 flat 544 markers placed on the topsurface of the body 564 are used by system 1004, to identify eachsubject outside of game surface tracking 584. In system 1004, whilefixed (X, Y) area tracking assemblies 504 are calibrated prior totracking, movable (X, Y, Z) volume tracking assemblies 506 arecalibrated simultaneously with tracking 592.

The preferred embodiment 1004 accomplishes the goals of real-time 3Dtracking without the limitations of currently available system 1002providing the following advantages:

1—By limiting the fixed (X, Y) area tracking matrix to a top view only,the system creates a scalable approach to camera placement that providesa substantially uniform pixel resolution per area;

2—By implementing a separate matrix of movable (X, Y, Z) volume trackingcameras to pick up the remaining side views of the players, the systemcreates a scalable approach to camera placement that provides asubstantially uniform pixel resolution per player;

3—By using either or both UV and IR light as the tracking energy that isemitted as a by-product from lamp sources that are also providingvisible lighting for general purposes, the system is both non-intrusiveand eye safe;

4—By using reflective as opposed to retroreflective markers the cone ofreflection is opened up such that separate ring lights are not requiredwhose added energy and visible light would be intrusive to both playersand audience;

5—By using markers that reflect only the narrow band of tracking energyand specifically do not reflect visible light, the marker's reflectionsare hidden from player and audience view;

6—By using flat markers that are embedded into the substrate the markeris no longer vulnerable to dislodge during player contact;

7—By using flat markers of non-uniform sizes and shapes the markers aremade visible for more camera angles thereby reducing the incidence ofinclusions;

8—By using flat markers of non-uniform sizes and shapes the markers canbe made to cover significantly larger surface area thereby reflectingmore of the tracking energy;

9—By using visibly transparent flat markers embedded into the substrateadding minimal additional thickness, the system's markers will now becompletely undetected by both the players and the audience;

10—By first strategically applying a combination of tracking energyabsorptive or reflective compounds to the background as well asforeground objects the system provides the ability for more clearlydistinguishing between background reflections, foreground objectreflections and marker reflections;

11—By separating player identification from the player joint trackingand isolating the identification marker to the top surface of theplayer, the system ensures a higher rate of player identification due tofewer inclusions of identification markers;

12—By separating player identification from the player joint trackingand isolating the identification marker to the top surface of theplayer, the system eliminates the importance of having substantially allbody joint markers in view at all times;

13—By separating player identification from the player joint trackingand isolating the identification marker to the top surface of theplayer, the system provides the possibility of performing playeridentification off the playing surface in the limited area of theentrance and exit and team benches where player movement is expected tobe significantly reduced thereby facilitating the identificationprocess;

14—By separating player identification from the player joint trackingand isolating the identification marker to the top surface of the playerthat can be zoomed in on while in the restricted movement areas of theentrance and exit and team benches, encoded markings similar to barcodes become feasible, which encode markings can easily handle forty ormore players and avoid any cumbersome pre-tracking procedure to ensureadequate marker “constellation” differentiation per player;

15—By establishing a separate fixed (X, Y) area tracking matrix that maybe pre-calibrated for the X and Y dimensions and by implementing fixedsize and shape relationship markers on rigid surfaces such as the playerhelmet and stick that can be used as calibration tools, the systemprovides dynamic calibration of movable cameras simultaneously withplayer tracking;

16—By providing dynamic calibration of movable cameras simultaneous withplayer tracking, the system further provides the ability to dynamicallyrecreate the optimal tracking volume for minimal marker inclusions;

17—By establishing a separate fixed (X, Y) area tracking matrix thatcontinually locates each player in (X, Y) space the system provides theability to automatically direct the pan, tilt and zoom aspects of one ormore movable cameras to follow each player; and

18—By establishing a system that dynamically follows each player withgreater accuracy and fewer marker inclusions, the system provides theability to predict more accurately the limited range of movement thatcould be expected from any player in the next instant, which abilityprovides a second method for automatically directing the pan, tilt andzoom aspects of one or more movable cameras to follow each player.

Referring now to FIG. 4, there is shown a block diagram substantiallysimilar to the preferred embodiment 1004 except that it employscomponents taught by the present inventors to provide a fluorescentalternative embodiment 1006. Embodiment 1006 specifically employsexisting lights emitting visible frequencies 524 that are absorbed byfluorescent 526 markers that in turn emit IR light 514 for the trackingfrequency 510. All other aspects and benefits of alternate embodiment1006 are identical to preferred embodiment 1004.

Embodiment 1006 accomplishes the goals of real-time 3D tracking, withoutthe limitations of system 1002, and provides the following additionaladvantages:

1—By using fluorescent markers that absorb in the visible region thesystem can rely fully upon existing rink lighting without modificationsto its emissions spectrum to supply the tracking energy, and

2—By using fluorescent markers that emit in the IR region the system canremain visually transparent to players and audience.

Referring now to FIG. 5, there is shown a block diagram substantiallysimilar to the preferred embodiment 1004 except that it employscomponents to provide a visibly transparent retroreflective alternativeembodiment 1008. Embodiment 1008 specifically employs visiblytransparent retroreflective 532 a markers in combination with ringlights 522 emitting either IR light 514 or UV light 516 trackingfrequencies 510. All other aspects and benefits of alternate embodiment1008 are identical to preferred embodiment 1004.

Embodiment 1008 accomplishes the goals of real-time 3D tracking, withoutthe limitations of system 1002, and provides the following additionaladvantage: using retroreflective markers that reflect only the narrowband of tracking frequencies of IR or UV, the system provides forgreater reflected signal strength while still remaining visiblytransparent to both players and audience.

Referring now to FIG. 6, there is shown a block diagram substantiallysimilar to the preferred embodiment 1004 except that it employscomponents to provide a game surface 1 d tracking alternative embodiment1010. Embodiment 1010 specifically uses fixed (X, Y) area trackingassemblies 504 to read the encoded marking 574 player id located on thetop surface of the body 564 during game surface tracking 582. All otheraspects and benefits of alternate embodiment 1010 are identical topreferred embodiment 1004.

Embodiment 1010 accomplishes the goals of real-time 3D tracking, withoutthe limitations of system 1002, and providing the following additionaladvantage: identifying players simultaneously with game surface trackingthe system provides the option of eliminating separate area trackingcameras in the non-playing surfaces of the entrance and exit passagewayand team benches.

Referring now to FIG. 7, there is shown a block diagram substantiallysimilar to the system 1002 except that it employs components to providea non-visible variation 1012. Variation 1012 specifically uses thevisibly transparent retroreflective markers 532 a first taught by thepresent inventors in their co-pending application rather thantraditional visibly retroreflective markers as currently used. Variation1012 further limits the tracking frequencies 510 to either IR light 514or UV light 516 and employs flat (embedded/applied ink) 544 non-uniformmulti-shape 554 markers.

Variation 1012 incrementally improves upon the real-time 3D trackingimplemented by system 1002 by providing the following additionaladvantages:

1—By using markers that reflect only the narrow band of tracking energyand specifically do not reflect visible light, the marker's reflectionsare hidden from player and audience view;

2—By using flat markers that are embedded into the substrate the markeris no longer vulnerable to dislodge during player contact;

3—By using flat markers of non-uniform sizes and shapes the markers aremade visible for more camera angles thereby reducing the incidence ofinclusions;

4—By using flat markers of non-uniform sizes and shapes the markers canbe made to cover significantly larger surface area thereby reflectingmore of the tracking energy, and

5—By using visibly transparent flat markers embedded into the substrateadding minimal thickness, the system's markers will now be completelyundetected by both the players and the audience.

Referring now to FIG. 8, there is shown a block diagram substantiallysimilar to the non-visible variation 1012 except that it employsadditional components to provide a top surface of body encoded idvariation 1014. Variation 1014 specifically uses flat 544 non-uniform554 top surface of body 564 encoded markings 574 to establish eachplayer's identification during game surface tracking 582.

Variation 1014 incrementally improves upon the real-time 3D trackingimplemented by non-visible variation 1012 by providing the followingadditional advantages:

1—By separating player identification from the player joint tracking andisolating the identification marker to the top surface of the player,the system ensures a higher rate of player identification due to fewerinclusions of identification markers,

2—By separating player identification from the player joint tracking andisolating the identification marker to the top surface of the player,the system eliminates the importance of having substantially all bodyjoint markers in view at all times, and

3—By separating player identification from the player joint tracking andisolating the identification marker to the top surface of the player,the system provides the possibility of performing player identificationoff the playing surface in the limited area of the entrance and exitpassageway and team benches where player movement is expected to besignificantly reduced thereby facilitating the identification process.

Referring now to FIG. 9, there is shown a block diagram substantiallysimilar to the top surface of body encoded id variation 1014 except thatit employs additional components to provide outside of game surface 1 dvariation 1016. Variation 1016 specifically establishes each player'sidentification outside of game surface tracking 584 using fixed (X, Y)area tracking assemblies 504 at least in these restricted areas.

Variation 1016 incrementally improves upon the real-time 3D trackingimplemented by top surface of body encoded id variation 1014 byproviding the following additional advantages:

1—By separating player identification into the outside of game surfaceareas such as the entrance and exit passageway and team benches, theidentification marker on the top surface of the player that can bezoomed in on without affecting body marker tracking while on the gamesurface. The possibility of zoomed fields of view for the fixed volumecameras in these special areas makes encoded markings similar to barcodes feasible. These encoded markings can easily handle forty or moreplayers and avoid any cumbersome pre-tracking procedure to ensureadequate marker “constellation” differentiation per player.

2—By using fixed (X, Y) area tracking cameras at least in theidentification areas of the entrance and exit passageway and teambenches, total camera use is made more efficient. The top-downorientation of the (X, Y) camera is better suited than the perspectiveorientation of the (X, Y, Z) camera for zoom-in viewing of the topsurface where the encoded markings are located.

Referring now to FIG. 10, there is shown a block diagram substantiallysimilar to outside of game surface 1 d variation 1016 except that itemploys additional components to provide existing light source variation1018. Variation 1018 specifically employs existing lights modified toemit non-visible frequencies 526 whose tracking energy is returned bythe markers using the reflective 534 Marker: Emission Method 530.

Variation 1018 incrementally improves upon the real-time 3D trackingimplemented by outside of game surface 1 d variation 1016 by providingthe following additional advantages:

1—By switching to reflective markers as opposed to retroreflective, thecone of reflected energy is greatly expanded and can now be thought ofas omni-directional thus eliminating the need to keep the camera's lensin close proximately to the emitting light source, and

2—By using existing lights as the tracking energy source, no additionalenergy is required and therefore added to the ambient lighting thatwould among other problems raise the temperature and add additionalproduction, installation and maintenance costs to the system.

Referring now to FIG. 11, there is shown a block diagram substantiallysimilar to existing light source variation 1018 except that it employsadditional components to provide fluorescent variation 1020. Variation1020 specifically employs existing lights emitting visible frequencies524 that are absorbed by fluorescent 526 markers that in turn emit IRlight 514 for the tracking frequency 510.

Variation 1020 incrementally improves upon the real-time 3D trackingimplemented by existing light source variation 1018 by providing thefollowing additional advantages:

1—By using fluorescent markers that absorb in the visible region thesystem can rely fully upon existing rink lighting without modificationsto its emissions spectrum to supply the tracking energy, and

2—By using fluorescent markers that emit in the IR region the system canremain visually transparent to players and audience.

Referring now to FIG. 12, there is shown a block diagram substantiallysimilar to outside of game surface 1 d variation 1016 except that itemploys additional components to provide movable volume trackingvariation 1022. Variation 1022 specifically employs movable (X, Y, Z)volume tracking assemblies 506 along with a calibration method 590 thatis performed simultaneously with tracking 594.

Variation 1022 incrementally improves upon the real-time 3D trackingimplemented by outside of game surface 1 d variation 1016 by providingthe following additional advantages:

1—Since both fixed area and fixed volume tracking assemblies arepre-calibrated prior to tracking, by adding pre-known markers to rigidsurfaces such as the player's helmet and stick or the boards and theirglass support columns, the system is now able to calibrate movablecameras simultaneously with tracking, and

2—By adding movable (X, Y, Z) volume tracking cameras that can remaincalibrated as they pan, tilt and zoom, the system can automaticallyaugment the combined FOV created by existing fixed volume trackingcameras whenever anticipated player bunching is expected to create anunacceptable level of marker inclusions.

In summary, FIG. 1 represents those components either in use withincurrently available systems such as Motion Analysis or Vicon or thecorresponding novel components taught by the present inventors.

At least the following components are considered to be novel and firsttaught by the present inventors for use within a multi-object trackingsystem:

1. fixed (X, Y) area tracking cameras 504 as a Camera Assemblies 500,

2. movable (X, Y, Z) volume tracking cameras 506 as a Camera Assemblies500,

3. UV light 516 as a Tracking Frequency 510,

4. existing lights emitting visible frequencies 524 as an Energy Source520,

5. existing lights modified to emit non-visible frequencies 526 as anEnergy Source 520,

6. retroreflective (visibly transparent) 532 b as an Emission Method fora Marker 530,

7. reflective (visibly transparent) 534 as an Emission Method for aMarker 530,

8. fluorescent (visibly transparent) 536 as an Emission Method for aMarker 530,

9. flat (embedded/applied ink) 544 as a Physical Form for a Marker 540,

10. non-uniform multi-shape 554 as a Reflective Shape for a Marker 550,

11. top surface of body 564 as a Location for the Identification 560,

12. encoded markings 574 as an Encoding Method for the Identification570,

13. outside of game surface tracking 584 as a time to Obtain theIdentification 580, and

14. simultaneously with tracking 594 as a time to perform theCalibration Method 590.

FIG. 2 represents the system 1002 that comprises a combination ofcomponents known and taught for use within a multi-object trackingsystem. FIG. 7 through FIG. 12 represent some of the possible and usefulvariations of the system 1002 including various of the additionalcomponents either first taught by the present inventors or firstconsidered for use within such multi-object tracking systems by thepresent inventors. FIG. 3 represents the preferred embodiment 1004 for amulti-object tracking system. FIG. 4 through FIG. 6 represent some ofthe possible and useful alternate compositions of the system 1002 inconsideration of known and taught components being applied in a novelway.

There are other novel components first taught by the present inventorsthat are not specifically identified in any of the FIGS. 1 through 12.Three such important components are:

1—the use of absorbers or reflectors to control the reflectivity ofbackground and foreground objects creating a clear distinction betweentheir detected energy intensities and that of the tracking markers,

2—the use of strategically placed markings on one or more rigid objectsmoving about with the players such as their helmet and stick in order toassist in the dynamic calibration of especially the movable volumetracking cameras, and

3—the use of strategically placed markings on one or more surfaces ofthe background such as the boards or their glass support columns toassist in the dynamic calibration of especially the movable volumetracking cameras.

The use of these components is considered to be important and equallyapplicable to any such system 1002 through 1022.

CONCLUSION, RAMIFICATIONS, AND SCOPE OF INVENTION

Thus the reader will see that the present invention successfully:

1—teaches the fundamental component groups necessary for a multi-objectreal-time 3D object tracking system;

2—identifies those individual components already in use within currentlyavailable systems and to which component groups they belong;

3—teaches those novel components suggested by the present inventors inthis and their other four co-pending applications and to which componentgroups they belong;

4—teaches how the novel components allow systems to better function in alive application with multiple colliding objects, for instance asporting event such as ice hockey;

5—identifies the composition of multi-object real-time 3Dobject-tracking systems in terms of actual components used from eachgroup;

6—teaches a novel preferred embodiment for a multi-object real-time 3Dobject-tracking system best suited for a live sporting event such as icehockey in terms of actual components used from each group;

7—teaches several novel alternative embodiments using one or morecomponents of the systems mixed into the preferred embodiment, and

8—teaches several novel variations using one or more of the novelcomponents mixed into the system.

Furthermore, the reader will also see that, for at least the preferredembodiment and to a great extant its alternates as well as thevariations of the systems, the present inventors have taught how toconstruct a system that:

1—is scalable and therefore comprises uniform assemblies that arecombinable into a matrix designed to increase tracking coverage in termsof area, volume or the number of objects while still maintaining uniformperformance;

2—is minimally intrusive upon the objects to be tracked and upon thesurrounding environment especially if that environment is a livesetting;

3—maximizes tracking signal-to-noise ratio;

4—minimizes manufacturing and installation costs, and

5—simplifies maintenance and operation for the user.

While the above description contains many details, these should not beconstrued as limitations on the scope of the invention, but rather as anexemplification of preferred embodiments thereof. Many aspects of thesystem's functionality are beneficial by themselves without otheraspects being present as will be appreciated by those skilled in theart. Furthermore, all of the novel combinations of components taughthave anticipated application beyond that of the tracking of livesporting events. Examples of other applications include but are notlimited to the tracking of human actors for the creation of animatedfilm sequences, the tracking of human subjects for medical research, aswell as other object tracking functions currently performed by existingsystems.

From the foregoing detailed description of the present invention, itwill be apparent that the invention has a number of advantages, some ofwhich have been described above and others that are inherent in theinvention. Also, it will be apparent that modifications can be made tothe present invention without departing from the teachings of theinvention. Accordingly, the scope of the invention is only to be limitedas necessitated by the accompanying claims.

What is claimed:
 1. A system for automatically providing event non-videoinformation usable for indexing event video information, where the eventtakes place at an event location for a duration of event time,comprising: a non-video information system for receiving or determiningevent non-video information, where the event non-video informationrelates to event activities taking place at the event location duringthe event time, where the event activities comprise the movements of oneor more event persons or event objects, and where the event non-videoinformation is relatable to the event time and is stored in a non-videodataset; a video information system for receiving event videoinformation captured by one or more filming cameras, where the eventvideo information records at least some of the event activities takingplace at the event location during the event time, and where the eventvideo information is relatable to event time and is stored in a videodataset; and a video information retrieval system for randomly accessingthe video dataset, where the video information retrieval system hasaccess to both the non-video dataset and the video dataset, where therandom access includes: (a) using event non-video information to firstdetermine an event time or duration of event time, and (b) using thedetermined event time or duration of event time to selectively access,retrieve, or provide event video information including one or more videoimages of the event.
 2. The system of claim 1 where the event non-videoinformation includes any one of, or any combination of eventquantifications relating to the event or one or more event persons orevent objects, where event quantifications include any one of, or anycombination of object tracking information or event statistics, wherethe object tracking information includes any one of, or any combinationof the on-going location, orientation, trajectory, acceleration orvelocity of one or more event persons or event objects, and where if theevent is a sporting event, event persons are athletes or game officials,event objects are game objects such as a puck in the sport of icehockey, and event statistics at least include any one of, or anycombination of: playing time, playing time per event sub-locations,average speed, time in control of the game object, number of scoringattempts, turnovers, steals or passes.
 3. The system of claim 2 whereone or more identifying items are affixed to one or more event personsor event objects prior to the event time, where the identifying itemsare usable for determining identifying item information that isnon-video information, and where the event quantifications aredetermined in part from the identifying item information.
 4. The systemof claim 3 where the one or more identifying items are marks detectableas video data, where the identifying item marks are represented withintracking video information captured by one or more tracking cameras,where the tracking video information is distinct from the event videoinformation, where the tracking video information is usable fordetermining the identifying item information by performing imageanalysis on the tracking video information in order to locate andquantify the identifying item information.
 5. The system of claim 4where the one or more identifying marks are any one of, or anycombination of non-visible or encoded, where encoded marks include barcodes or similar formats.
 6. The system of claim 3 where the objecttracking information determined from the identifying item informationincludes three dimensional representations of one or more event personsor event objects, where the three dimensional representations are usedto generate event animation information, and where the event videoinformation stored in the video dataset is any combination of the eventvideo information captured from cameras and the event animationinformation generated from the object tracking information.
 7. Thesystem of claim 6 where the video information retrieval system isfurther enabled to retrieve any combination of the event videoinformation, event animation information or event non-video informationeither local to the event or remote to the event, where the time ofaccess is either during the event time or after the event time, andwhere the time of access during the event time is either real-time orpost real-time.
 8. The system of claim 2 where the event non-videoinformation comprises in part object tracking information, where theobject tracking information is determined in part by performing imageanalysis on tracking video information captured from one or moretracking cameras, where the tracking video information is distinct fromthe event video information and includes event activities beingperformed during event time.
 9. The system of claim 8 where the trackingvideo information is usable as event video information.
 10. The systemof claim 8 where one or more identifying marks are affixed to one ormore event persons or event objects prior to the event time, where theidentifying marks are usable for determining identifying markinformation that is non-video information, where the identifying marksare detectable as video data represented within the tracking videoinformation, where the tracking video information is usable fordetermining the identifying mark information by performing imageanalysis on the tracking video information in order to locate andquantify the identifying mark information, and where the eventquantifications are determined in part from the identifying markinformation.
 11. The system of claim 1 where the event non-videoinformation includes pre-event non-video information determined prior tothe duration of event time and live-event non-video informationdetermined during the duration of event time, where the pre-eventnon-video information includes event person group information or eventlocation information, where the event location information includeslayout information describing the event area where the event activitiestake place, where the event is a sporting event and the person groupinformation includes a list of players and team associations if thesporting event is a team sport, and where the live-event non-videoinformation includes any one of, or any combination of the eventquantifications.
 12. The system of claim 1 where the non-videoinformation system, the video information system and the videoinformation retrieval system operate in real-time such that the randomaccess of the event video information occurs either in real-time or postreal-time.
 13. The system of claim 1 where individuals access andretrieve any one of, or any combination of the event video informationand the event non-video information either local to the event or remoteto the event, where the time of access is either during the event timeor after the event time, and where the time of access during the eventtime is either real-time or post real-time.
 14. The system of claim 1where any one of, or any combination of: 1) the non-video informationsystem, 2) the video information system, or 3) the video informationretrieval system, comprise either different systems or the same system.15. The system of claim 2 where at least one of the one or more filmingcameras for capturing event video information are adjustable filmingcameras, where the adjustments include any one of, or any combination ofchanges in the pan, tilt or zoom setting of the adjustable filmingcamera, where each adjustable filming camera is further adapted toreceive electronic adjustment signals and to make correspondingadjustments, and where the electronic adjustment signals are generatedin response to the event quantifications.
 16. The system of claim 15where event quantifications including object tracking data are used toassign or re-assign any adjustable filming camera to film any eventpersons.
 17. The system of claim 1 where the event video informationsystem has access to event location background image informationcorresponding to the captured event video information, where the eventvideo information system compares the captured event video informationto the corresponding background image information in order to create aminimal set of foreground pixels, and where the event video informationstored in the video dataset is limited based at least in part upon thedetermined minimal set of foreground pixels.
 18. The system of claim 17where the video information retrieval system has access to thebackground image information corresponding to the video dataset offoreground limited event video information, where the video informationretrieval system recreates non-limited event video informationcomprising the minimal set of foreground pixels combined with thecorresponding background image information.
 19. The system of claim 17where the event video information system analyzes the minimal set offoreground pixels to create a stylized graphical representation of theevent foreground, and where the event video information stored in thevideo dataset includes the stylized graphical representation.
 20. Thesystem of claim 17 where the video information retrieval system presentsa graphical representation of the randomly accessed event videoinformation stored in the video dataset using at least in part thestylized graphical representations of the event foreground containedwithin the video dataset.
 21. A system for allowing a user tointeractively retrieve event video information by first randomlyaccessing event non-video information, comprising: a data synchronizeroperative on a computer for receiving or accessing event non-videoinformation and event video information, where the data synchronizercross-links the event non-video dataset with the event video datasetusing event time such that the event non-video information is relatableto the event video information by event time, and a video informationretrieval system operative on a computer for randomly accessing theevent video information using the synchronized non-video information,where the random access includes: (a) using event non-video informationto first determine an event time or duration of event time, and (b)using the determined event time or duration of event time to selectivelyaccess, retrieve, or provide event video information including one ormore video images of the event.
 22. The system of claim 21 where theevent non-video information includes any one of, or any combination ofevent quantifications relating to the event or one or more event personsor event objects, where event quantifications include any one of, or anycombination of object tracking information or event statistics, wherethe object tracking information includes any one of, or any combinationof the on-going location, orientation, trajectory, acceleration orvelocity of one or more event persons or event objects, and where if theevent is a sporting event, event persons are athletes or game officials,event objects are game objects such as a puck in the sport of icehockey, and event statistics at least include any one of, or anycombination of: playing time, playing time per event sub-locations,average speed, time in control of the game object, number of scoringattempts, turnovers, steals or passes.
 23. The system of claim 22 whereone or more identifying items are affixed to one or more event personsor event objects prior to the event time, where the identifying itemsare usable for determining identifying item information that isnon-video information, where the event quantifications are determined inpart from the identifying item information, and where the datasynchronizer additionally receives or accesses the identifying iteminformation as additional non-video information.
 24. The system of claim22 where the event non-video information includes pre-event non-videoinformation determined prior to the duration of event time andlive-event non-video information determined during the duration of eventtime, where the pre-event non-video information includes event persongroup information or event location information, where the eventlocation information includes layout information describing the eventarea where the event activities take place, where the event is asporting event and the person group information includes a list ofplayers and team associations if the sporting event is a team sport, andwhere the live-event non-video information includes any one of, or anycombination of the event quantifications.
 25. The system of claim 21where the data synchronizer and the video information retrieval systemoperate in real-time such that the random access of the event videoinformation occurs either in real-time or post real-time.
 26. A systemfor creating a non-video dataset in accordance with an event includingevent non-video information relatable to event time for use in providingrandom access to a video dataset captured in accordance with the eventincluding event video information relatable to event time, comprising: anon-video information system for receiving or determining eventnon-video information, where the event non-video information relates toevent activities taking place at an event location during event time,where the event activities comprise the movements of one or more eventpersons or event objects, and a non-video dataset, where the non-videodataset includes the event non-video information stored in relation toevent time, where the non-video dataset is relatable to a video datasetstored in relation to event time, where the video dataset includes eventvideo information, where the event video information is captured by oneor more filming cameras, where the event video information records atleast some of the event activities taking place at the event locationduring the event time and is relatable to event time, where thenon-video data is adapted to be used by a video information retrievalsystem in order to randomly access the video dataset, where the videoinformation retrieval system has access to both the non-video datasetand the video dataset, and where the random access includes: (a) usingevent non-video information to first determine an event time or durationof event time, and (b) using the determined event time or duration ofevent time to selectively access, retrieve, or provide event videoinformation including one or more video images of the event.
 27. Thesystem of claim 26 where the event non-video information includes anyone of, or any combination of event quantifications relating to theevent or one or more event persons or event objects, where eventquantifications include any one of, or any combination of objecttracking information or event statistics, where the object trackinginformation includes any one of or any combination of the on-goinglocation, orientation, trajectory, acceleration or velocity of one ormore event persons or event objects, and where if the event is asporting event, event persons are athletes or game officials, eventobjects are game objects such as a puck in the sport of ice hockey, andevent statistics at least include any one of, or any combination of:playing time, playing time per event sub-locations, average speed, timein control of the game object, number of scoring attempts, turnovers,steals or passes.
 28. The system of claim 27 where one or moreidentifying items are affixed to one or more event persons or eventobjects prior to the event time, where the identifying items are usablefor determining identifying item information that is non-videoinformation, where the event quantifications are determined in part fromthe identifying item information, and where the video informationretrieval system additionally receives or accesses the identifying iteminformation as additional non-video information.
 29. The system of claim26 where the event non-video information includes pre-event non-videoinformation determined prior to the duration of event time andlive-event non-video information determined during the duration of eventtime, where the pre-event non-video information includes event persongroup information or event location information, where the eventlocation information includes layout information describing the eventarea where the event activities take place, where the event is asporting event and the person group information includes a list ofplayers and team associations if the sporting event is a team sport, andwhere the live-event non-video information includes any one of, or anycombination of the event quantifications.
 30. The system of claim 26where the non-video information system and the video informationretrieval system operate in real-time such that the random access of theevent video information occurs either in real-time or post real-time.